Nov 02 08:03:12.874979 ip-10-0-139-247 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 08:03:12.874990 ip-10-0-139-247 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 08:03:12.874997 ip-10-0-139-247 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 08:03:12.875350 ip-10-0-139-247 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 08:03:22.906635 ip-10-0-139-247 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 08:03:22.906651 ip-10-0-139-247 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7822187e731444a0b29dc9def9634be1 -- Nov 02 08:05:10.180571 ip-10-0-139-247 systemd[1]: Starting Kubernetes Kubelet... Nov 02 08:05:10.583472 ip-10-0-139-247 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:10.583472 ip-10-0-139-247 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 08:05:10.583472 ip-10-0-139-247 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:10.583472 ip-10-0-139-247 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:10.586802 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586725 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.586802 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586802 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.586802 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586806 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586809 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586812 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586815 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586819 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586822 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586824 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586827 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586830 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586833 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586836 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586838 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586846 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586850 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586853 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586856 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586858 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586861 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586864 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.586892 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586867 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586869 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586872 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586875 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586877 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586880 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586883 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586886 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586888 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586891 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586894 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586896 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586899 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586902 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586906 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586909 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586911 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586914 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586917 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.587352 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586920 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586923 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586926 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586929 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586933 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586936 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586939 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586942 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586944 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586947 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586950 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586953 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586956 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586958 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586960 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586964 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586967 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586969 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586972 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.587797 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586975 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586978 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586980 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586983 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586986 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586989 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586991 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586994 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586996 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.586999 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587002 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587004 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587007 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587010 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587013 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587015 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587018 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587020 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587023 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587025 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.588258 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587028 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587031 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587033 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587036 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587038 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587041 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587043 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587046 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587049 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587052 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587054 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587057 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587060 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587062 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587065 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587068 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587070 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587073 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587077 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587080 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.588749 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587084 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587088 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587091 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587094 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587097 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587099 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587103 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587895 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587902 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587905 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587908 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587911 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587913 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587916 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587919 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587921 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587924 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587927 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587929 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.589211 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587932 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587936 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587939 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587942 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587945 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587947 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587950 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587953 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587956 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587958 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587961 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587963 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587966 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587969 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587972 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587974 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587977 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587979 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587982 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587985 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.589758 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587988 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587990 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587993 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587996 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.587999 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588001 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588004 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588006 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588009 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588011 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588014 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588017 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588020 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588023 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588026 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588028 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588031 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588034 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588037 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588040 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.590237 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588042 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588045 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588048 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588050 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588053 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588057 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588061 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588064 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588067 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588070 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588073 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588077 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588081 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588085 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588088 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588092 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588094 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588097 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588100 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.590768 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588102 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588105 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588108 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588111 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588113 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588116 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588119 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588122 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588125 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588128 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588130 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588133 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588136 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588138 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588141 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588143 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588146 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588149 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588151 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588153 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.591227 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588156 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588159 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588161 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588164 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588166 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588169 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588171 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588174 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588177 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588179 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588182 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588184 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588187 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588190 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588194 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588702 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588707 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588711 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588713 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588716 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.591727 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588719 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588722 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588725 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588728 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588730 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588733 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588736 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588739 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588741 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588744 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588746 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588749 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588751 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588755 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588757 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588760 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588763 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588765 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588768 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588771 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.592204 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588774 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588777 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588780 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588784 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588787 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588790 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588797 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588800 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588802 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588805 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588808 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588811 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588813 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588816 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588819 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588821 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588824 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588827 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588829 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.592682 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588834 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588838 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588841 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588844 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588846 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588849 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588852 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588854 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588857 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588860 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588863 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588865 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588868 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588871 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588874 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588877 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588880 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588883 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588885 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.593130 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588891 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588893 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588896 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588899 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588901 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588904 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588907 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588909 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588912 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588915 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588917 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588920 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588923 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588925 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588928 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588930 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588933 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588936 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588938 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588941 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.593590 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588944 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588946 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588949 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588951 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588954 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588956 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588959 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588962 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588965 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588967 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588970 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588972 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588977 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588980 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588982 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588985 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588987 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588990 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588993 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588995 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.594066 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.588998 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.589000 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.589004 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589748 2582 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589759 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589765 2582 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589770 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589774 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589778 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589782 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589786 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589790 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589793 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589796 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589799 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589803 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589806 2582 flags.go:64] FLAG: --cgroup-root="" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589809 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589812 2582 flags.go:64] FLAG: --client-ca-file="" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589815 2582 flags.go:64] FLAG: --cloud-provider="external" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589819 2582 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589823 2582 flags.go:64] FLAG: --cluster-domain="" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589826 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 08:05:10.594558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589830 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589836 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589839 2582 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589843 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589847 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589850 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589853 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589856 2582 flags.go:64] FLAG: --contention-profiling="false" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589859 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589862 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589865 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589868 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589873 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589876 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589879 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589882 2582 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589886 2582 flags.go:64] FLAG: --enable-server="true" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589889 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589894 2582 flags.go:64] FLAG: --event-burst="100" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589897 2582 flags.go:64] FLAG: --event-qps="50" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589900 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589903 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589906 2582 flags.go:64] FLAG: --eviction-hard="" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589910 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 08:05:10.595088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589913 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589916 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589919 2582 flags.go:64] FLAG: --eviction-soft="" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589923 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589926 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589929 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589932 2582 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589935 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589938 2582 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589943 2582 flags.go:64] FLAG: --feature-gates="" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589947 2582 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589951 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589954 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589957 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589961 2582 flags.go:64] FLAG: --healthz-port="10248" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589964 2582 flags.go:64] FLAG: --help="false" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589967 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589970 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589974 2582 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589976 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589980 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589983 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589986 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 08:05:10.595724 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589989 2582 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589992 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589995 2582 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.589998 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590001 2582 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590004 2582 flags.go:64] FLAG: --kube-reserved="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590007 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590010 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590013 2582 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590016 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590019 2582 flags.go:64] FLAG: --lock-file="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590021 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590025 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590028 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590033 2582 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590036 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590040 2582 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590043 2582 flags.go:64] FLAG: --logging-format="text" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590047 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590051 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590054 2582 flags.go:64] FLAG: --manifest-url="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590057 2582 flags.go:64] FLAG: --manifest-url-header="" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590067 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590070 2582 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590077 2582 flags.go:64] FLAG: --max-pods="110" Nov 02 08:05:10.596359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590080 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590083 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590086 2582 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590089 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590092 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590095 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590098 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590106 2582 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590109 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590112 2582 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590115 2582 flags.go:64] FLAG: --pod-cidr="" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590118 2582 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590121 2582 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590124 2582 flags.go:64] FLAG: --pods-per-core="0" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590127 2582 flags.go:64] FLAG: --port="10250" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590130 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590133 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-032ef74f3e0e41b96" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590136 2582 flags.go:64] FLAG: --qos-reserved="" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590139 2582 flags.go:64] FLAG: --read-only-port="10255" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590142 2582 flags.go:64] FLAG: --register-node="true" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590145 2582 flags.go:64] FLAG: --register-with-taints="" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590149 2582 flags.go:64] FLAG: --registry-burst="10" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590152 2582 flags.go:64] FLAG: --registry-qps="5" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590154 2582 flags.go:64] FLAG: --reserved-cpus="" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590158 2582 flags.go:64] FLAG: --reserved-memory="" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590163 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 08:05:10.596975 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590167 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590170 2582 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590173 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590176 2582 flags.go:64] FLAG: --runonce="false" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590179 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590183 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590185 2582 flags.go:64] FLAG: --seccomp-default="false" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590188 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590192 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590195 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590198 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590201 2582 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590204 2582 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590207 2582 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590210 2582 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590213 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590216 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590219 2582 flags.go:64] FLAG: --system-cgroups="" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590222 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590227 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590230 2582 flags.go:64] FLAG: --tls-cert-file="" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590233 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590237 2582 flags.go:64] FLAG: --tls-min-version="" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590240 2582 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590254 2582 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 08:05:10.597607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590258 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590261 2582 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590264 2582 flags.go:64] FLAG: --v="2" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590269 2582 flags.go:64] FLAG: --version="false" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590273 2582 flags.go:64] FLAG: --vmodule="" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590278 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590284 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590425 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590430 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590434 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590438 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590442 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590444 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590447 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590450 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590452 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590455 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590458 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590460 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590463 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590466 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.598343 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590468 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590471 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590474 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590477 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590479 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590482 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590484 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590487 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590490 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590492 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590495 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590498 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590500 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590503 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590506 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590508 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590511 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590516 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590520 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590523 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.599148 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590525 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590528 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590531 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590533 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590536 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590539 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590541 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590544 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590547 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590549 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590552 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590555 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590557 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590560 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590563 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590565 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590568 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590571 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590573 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.600022 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590576 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590579 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590581 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590584 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590586 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590589 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590592 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590594 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590597 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590600 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590605 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590608 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590611 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590614 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590616 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590619 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590621 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590624 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590627 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590629 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.600843 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590632 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590636 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590640 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590643 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590646 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590649 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590652 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590655 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590658 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590661 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590664 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590667 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590669 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590672 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590674 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590677 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590680 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590682 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590685 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590688 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.601491 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590690 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590693 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590699 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590701 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590704 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590707 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590710 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590713 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590715 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590718 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590721 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590724 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590726 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.590753 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590787 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.602097 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590791 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590795 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590797 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590801 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590804 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590808 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590810 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590813 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590816 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590819 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590821 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590824 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590827 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590829 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590832 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590835 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590838 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590840 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590843 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.602489 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590847 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590851 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590854 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590856 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590860 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590863 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590865 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590868 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590871 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590873 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590876 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590879 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590882 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590884 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590887 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590889 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590892 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590895 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590898 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590900 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.602942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590904 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590906 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590909 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590912 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590914 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590918 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590922 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590924 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590927 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590930 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590932 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590935 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590939 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590943 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590947 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590950 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590953 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590956 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590958 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.603484 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590961 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590964 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590967 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590969 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590972 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590975 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590977 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590980 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590983 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590985 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590988 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590991 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590993 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590996 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.590998 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591001 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591004 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591006 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591009 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.603933 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591012 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591015 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591018 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591022 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591025 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591028 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591032 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591036 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591038 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591041 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591044 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591046 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591050 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591052 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591056 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591059 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591062 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591064 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591067 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591070 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.604704 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591073 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591075 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591078 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591080 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591083 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591085 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591088 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.591091 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.598123 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.598147 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.598166 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.598174 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598363 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598372 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598376 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598381 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598385 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598390 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598394 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.605460 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598399 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598403 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598414 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598418 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598423 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598427 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598432 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598436 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598441 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598445 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598449 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598456 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598461 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598467 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598477 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598481 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598485 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598492 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.606063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598499 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598504 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598508 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598513 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598520 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598525 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598530 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598534 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598539 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598549 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598553 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598558 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598562 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598566 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598571 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598575 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598581 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598585 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598590 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.606565 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598594 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598598 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598602 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598612 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598616 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598621 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598625 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598630 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598635 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598639 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598644 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598648 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598653 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598657 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598661 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598671 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598675 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598680 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598685 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.607017 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598690 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598694 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598699 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598703 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598708 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598712 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598716 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598720 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598725 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598733 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598738 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598742 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598746 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598751 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598755 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598759 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598764 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598768 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598772 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598776 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598781 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.607586 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598785 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598794 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598799 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598803 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598807 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598812 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598816 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598820 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598824 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598829 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598834 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598839 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598844 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598854 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598858 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598863 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598868 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598872 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598876 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.608113 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598880 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598885 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.598889 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.598956 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599448 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599464 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599470 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599475 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599486 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599491 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599496 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599500 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599506 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599510 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599514 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.608891 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599519 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599523 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599527 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599532 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599537 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599541 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599546 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599550 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599554 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599559 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599564 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599568 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599572 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599576 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599580 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599585 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599589 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599593 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599598 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599602 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.609558 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599606 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599611 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599616 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599621 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599625 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599629 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599634 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599638 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599642 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599647 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599651 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599656 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599660 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599664 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599669 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599674 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599678 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599682 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599687 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599691 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.610377 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599696 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599700 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599704 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599709 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599713 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599717 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599722 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599726 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599730 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599734 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599738 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599742 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599750 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599756 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599762 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599767 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599771 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599776 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599780 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599784 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.610917 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599790 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599794 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599799 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599803 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599807 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599811 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599816 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599820 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599824 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599828 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599833 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599837 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599841 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599845 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599849 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599854 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599858 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599862 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599866 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.611502 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599871 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599875 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599879 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599883 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599888 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599898 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599902 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599906 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599910 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599915 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599919 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599923 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599928 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599933 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.599937 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.611942 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.599985 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600038 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600046 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600051 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600056 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600060 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600065 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600069 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600074 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600080 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600088 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600093 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600097 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600102 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600106 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600110 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600115 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600119 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600123 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:10.612350 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600128 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600132 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600136 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600141 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600146 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600150 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600155 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600159 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600163 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600167 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600172 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600176 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600181 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600186 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600190 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600194 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600198 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600202 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600207 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600211 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:10.612912 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600215 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600219 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600223 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600228 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600232 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600236 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600241 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600261 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600265 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600271 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600275 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600279 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600283 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600288 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600292 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600297 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600301 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600306 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600310 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600315 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:10.613547 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600319 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600324 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600328 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600332 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600337 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600342 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600346 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600351 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600355 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600360 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600364 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600368 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600372 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600379 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600384 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600389 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600393 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600398 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600402 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600406 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:10.614063 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600410 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600415 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600418 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600422 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600426 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600431 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600437 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600441 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600446 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600451 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600456 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600462 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600466 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600471 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600475 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600479 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600483 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600488 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600493 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:10.614552 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600497 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600501 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600505 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600509 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600513 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600518 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600523 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600528 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:10.600532 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.600619 2582 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.605593 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.606465 2582 server.go:1014] "Starting client certificate rotation" Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.606568 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.606607 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.608684 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:10.614998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.610514 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:10.623388 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.623372 2582 log.go:25] "Validated CRI v1 runtime API" Nov 02 08:05:10.628169 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.628150 2582 log.go:25] "Validated CRI v1 image API" Nov 02 08:05:10.629019 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.628987 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 08:05:10.632631 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.632602 2582 fs.go:135] Filesystem UUIDs: map[34235341-2beb-495d-94d8-8db3bd8fa51a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c345fb7-3f99-4983-97a0-0f6a908df7f3:/dev/nvme0n1p4] Nov 02 08:05:10.632694 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.632630 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 08:05:10.639077 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.638961 2582 manager.go:217] Machine: {Timestamp:2026-11-02 08:05:10.637378223 +0000 UTC m=+0.341834027 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101424 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c491031534c948e4e908ad82939a SystemUUID:ec20c491-0315-34c9-48e4-e908ad82939a BootID:7822187e-7314-44a0-b29d-c9def9634be1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d7:bc:18:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d7:bc:18:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:26:bb:1a:de:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 08:05:10.639166 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.639079 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 08:05:10.639204 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.639161 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 08:05:10.642167 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.642146 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 08:05:10.642320 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.642168 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-247.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 08:05:10.642366 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.642322 2582 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 08:05:10.642366 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.642331 2582 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 08:05:10.642366 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.642359 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:05:10.642447 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.642379 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:05:10.643304 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.643294 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 08:05:10.647672 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.647661 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 08:05:10.647794 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.647785 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 08:05:10.648469 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.648460 2582 kubelet.go:493] "Attempting to sync node with API server" Nov 02 08:05:10.648497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.648472 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 08:05:10.648497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.648483 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 08:05:10.648497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.648492 2582 kubelet.go:405] "Adding apiserver pod source" Nov 02 08:05:10.648582 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.648499 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 08:05:10.649123 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.649111 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:05:10.652488 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.652070 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 08:05:10.656810 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.656794 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:05:10.657734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.657719 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 08:05:10.657816 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.657749 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 08:05:10.658404 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.658391 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 08:05:10.658404 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.658407 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 08:05:10.658502 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.658414 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 08:05:10.658502 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.658420 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 08:05:10.658502 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.658426 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 08:05:10.658502 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.658432 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 08:05:10.659021 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659013 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 08:05:10.659049 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659022 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 08:05:10.659049 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659029 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 08:05:10.659049 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659036 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 08:05:10.659049 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659045 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 08:05:10.659286 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659053 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 08:05:10.659286 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659075 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 08:05:10.659286 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.659081 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 08:05:10.662752 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.662741 2582 server.go:1263] "Started kubelet" Nov 02 08:05:10.662918 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.662870 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 08:05:10.662995 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.662865 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 08:05:10.663029 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.663007 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 08:05:10.663489 ip-10-0-139-247 systemd[1]: Started Kubernetes Kubelet. Nov 02 08:05:10.664134 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.664118 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 08:05:10.664610 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.664594 2582 server.go:317] "Adding debug handlers to kubelet server" Nov 02 08:05:10.670811 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.670794 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 08:05:10.671398 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.670876 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 08:05:10.671398 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.670955 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" Nov 02 08:05:10.671398 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.671195 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 08:05:10.671557 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.671542 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 08:05:10.671606 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.671544 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 08:05:10.671606 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.671575 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 08:05:10.671740 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.671723 2582 reconstruct.go:97] "Volume reconstruction finished" Nov 02 08:05:10.671740 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.671736 2582 reconciler.go:29] "Reconciler: start to sync state" Nov 02 08:05:10.675875 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.675856 2582 factory.go:55] Registering systemd factory Nov 02 08:05:10.675875 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.675877 2582 factory.go:223] Registration of the systemd container factory successfully Nov 02 08:05:10.676090 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.676076 2582 factory.go:153] Registering CRI-O factory Nov 02 08:05:10.676090 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.676091 2582 factory.go:223] Registration of the crio container factory successfully Nov 02 08:05:10.676194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.676139 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 08:05:10.676194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.676154 2582 factory.go:103] Registering Raw factory Nov 02 08:05:10.676194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.676166 2582 manager.go:1199] Started watching for new ooms in manager Nov 02 08:05:10.677199 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.677180 2582 manager.go:319] Starting recovery of all containers Nov 02 08:05:10.687491 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.687461 2582 manager.go:324] Recovery completed Nov 02 08:05:10.692528 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.692516 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.694386 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694368 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.694475 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694397 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.694475 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694408 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.694953 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694939 2582 cpu_manager.go:226] "Starting" policy="none" Nov 02 08:05:10.695005 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694952 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 08:05:10.695005 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694939 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.695005 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.694982 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 08:05:10.695771 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.695750 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.695863 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.695785 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.695863 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.695798 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.697149 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.697132 2582 policy_none.go:50] "Start" Nov 02 08:05:10.697211 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.697153 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 08:05:10.697211 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.697167 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 08:05:10.699265 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.699241 2582 policy_none.go:44] "Start" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.733275 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.735833 2582 manager.go:341] "Starting Device Plugin manager" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.735921 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.735933 2582 server.go:87] "Starting device plugin registration server" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.736212 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.736223 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.736500 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.736581 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.736590 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.737030 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.737067 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.738783 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.739914 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.739934 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.739954 2582 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 08:05:10.748637 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.739984 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 08:05:10.836769 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.836696 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.840178 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.840158 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal"] Nov 02 08:05:10.840274 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.840215 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.840512 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.840493 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.840563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.840535 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.840563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.840550 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.840636 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.840579 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.843766 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.843749 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.843867 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.843777 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.843867 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.843788 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.846004 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.845988 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.846110 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.846156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846127 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.846643 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846628 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.846705 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846660 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.846705 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846670 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.846705 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846630 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.846788 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846722 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.846788 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.846732 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.848702 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.848689 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.848747 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.848714 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:10.849353 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.849328 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:10.849429 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.849362 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:10.849429 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.849371 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:10.862659 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.862634 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-247.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 08:05:10.862763 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.862733 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-247.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 08:05:10.863065 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.863051 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 08:05:10.863119 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.863106 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgs8v" Nov 02 08:05:10.863608 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.862652 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-247.us-west-2.compute.internal.18e42bcce7cebc41 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-247.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-247.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 08:05:10.662708289 +0000 UTC m=+0.367164094,LastTimestamp:2026-11-02 08:05:10.662708289 +0000 UTC m=+0.367164094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-247.us-west-2.compute.internal,}" Nov 02 08:05:10.872709 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.872683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34d28ca7a91586006c95d24bde05028b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"34d28ca7a91586006c95d24bde05028b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.872709 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.872707 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.872840 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.872725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.878102 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.878080 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.884378 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:10.884360 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.969493 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.969454 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:10.969741 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.969536 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:10.972128 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.972111 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:10.972983 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.972967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34d28ca7a91586006c95d24bde05028b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"34d28ca7a91586006c95d24bde05028b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.973030 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.972991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.973030 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.973008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.973103 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.973041 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.973103 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.973053 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34d28ca7a91586006c95d24bde05028b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"34d28ca7a91586006c95d24bde05028b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:10.973103 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:10.973053 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:11.070592 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:11.070429 2582 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-247.us-west-2.compute.internal\" not found" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:11.071600 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.071582 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:11.167847 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:11.167765 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-247.us-west-2.compute.internal\" not found" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:11.181693 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.181673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:11.188080 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.188065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:11.271545 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.271513 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:11.272900 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.272877 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:11.273078 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.272935 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:11.273078 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.272951 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:11.273195 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.273090 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:11.328811 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:11.328780 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:12.494405 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.494365 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:12.554643 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.554554 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:12.555483 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.555470 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:12.666707 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.666679 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:12.729154 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.729115 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 08:05:12.729624 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.729275 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 08:05:12.729624 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.729276 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 08:05:12.729624 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.729309 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 08:05:12.729624 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:12.729334 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4c4bab8d27dd4bf6a9cd51fc9ec0bbf-5d8bc6b0cc418bd2.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-247.us-west-2.compute.internal.18e42bcce9b28abd\": read tcp 10.0.139.247:46612->3.141.133.146:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-247.us-west-2.compute.internal.18e42bcce9b28abd default 4763 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-247.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-247.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-247.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 08:05:10 +0000 UTC,LastTimestamp:2026-11-02 08:05:10.840555989 +0000 UTC m=+0.545011806,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-247.us-west-2.compute.internal,}" Nov 02 08:05:12.772215 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.772186 2582 apiserver.go:52] "Watching apiserver" Nov 02 08:05:12.794832 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.794799 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 08:05:12.990889 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.990804 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 08:00:11 +0000 UTC" deadline="2028-08-06 17:34:52.246719954 +0000 UTC" Nov 02 08:05:12.990889 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:12.990833 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15441h29m39.255890854s" Nov 02 08:05:13.042478 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.042445 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 08:05:13.042758 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.042742 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2sz6m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd","openshift-multus/multus-additional-cni-plugins-phvgd","openshift-network-diagnostics/network-check-target-8d2bg","kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-nldh5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-2phd8","openshift-dns/node-resolver-m29jr","openshift-multus/multus-g4xwk","kube-system/konnectivity-agent-hlm8n","openshift-image-registry/node-ca-s52jj","openshift-multus/network-metrics-daemon-trgc8","openshift-network-operator/iptables-alerter-vc95q"] Nov 02 08:05:13.045294 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.045277 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.047367 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.047346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.049492 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.049472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.051548 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.051533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:13.051620 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.051585 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:13.053662 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.053646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.055873 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.055854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.055941 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.055923 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:13.058024 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.058008 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.060223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.060206 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.062209 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.062194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.064219 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.064204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.066444 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.066427 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.066516 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.066496 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:13.068558 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.068542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.095738 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.095716 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 08:05:13.098286 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.098246 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 08:05:13.098286 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.098263 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 08:05:13.098470 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.098287 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 08:05:13.098470 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.098296 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.098470 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.098360 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zgn8x\"" Nov 02 08:05:13.098912 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.098868 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.099227 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.099209 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 08:05:13.099764 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.099743 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.099954 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.099936 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwch9\"" Nov 02 08:05:13.100103 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.100087 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.101319 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.100396 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 08:05:13.101519 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.101441 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 08:05:13.101672 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.101649 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 08:05:13.101853 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.101836 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-shlrq\"" Nov 02 08:05:13.101979 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.101961 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 08:05:13.102036 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.102023 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.103822 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-conf\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.103907 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-kubelet\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.103907 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.103907 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-dbus\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.104057 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-bin\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104057 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cni-binary-copy\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.104057 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.103992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-bin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.104057 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-netns\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104226 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-var-lib-kubelet\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.104226 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-slash\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104285 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104226 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-netd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104285 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104245 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.104285 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggvl\" (UniqueName: \"kubernetes.io/projected/ce7abe8e-c9a4-4128-a744-0a2cd073c535-kube-api-access-7ggvl\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.104285 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-sys-fs\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.104394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104291 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-env-overrides\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.104394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104370 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-systemd-units\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-var-lib-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104505 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-etc-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104505 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-systemd\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.104505 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-lib-modules\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.104505 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104483 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5hl\" (UniqueName: \"kubernetes.io/projected/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-kube-api-access-4h5hl\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.104613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-etc-kubernetes\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.104613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-kubelet\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104568 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9r2w\" (UniqueName: \"kubernetes.io/projected/d38b2552-896f-498f-9719-5aa7c76561a2-kube-api-access-h9r2w\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.104613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.104721 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.104721 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-kubernetes\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.104721 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmhs\" (UniqueName: \"kubernetes.io/projected/506e21c2-5241-44f6-8c30-0581ebe68600-kube-api-access-gsmhs\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.104807 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-netns\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.104807 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104761 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-conf-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.104807 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-systemd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104892 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-log-socket\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.104892 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104829 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-system-cni-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.104892 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-modprobe-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.104892 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-kubelet-config\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.104892 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-socket-dir-parent\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105047 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-socket-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.105047 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-registration-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.105047 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.104988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-tmp\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105047 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105015 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-multus-certs\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105047 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rz5\" (UniqueName: \"kubernetes.io/projected/89367e5b-7b56-4918-95f3-5ae7c2ad5293-kube-api-access-c5rz5\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznrq\" (UniqueName: \"kubernetes.io/projected/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-kube-api-access-tznrq\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-run\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-host\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-k8s-cni-cncf-io\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105223 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105204 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-daemon-config\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-config\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506e21c2-5241-44f6-8c30-0581ebe68600-tmp-dir\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-system-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468nc\" (UniqueName: \"kubernetes.io/projected/c2ebff0e-2c89-4345-8577-9e296f299461-kube-api-access-468nc\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38b2552-896f-498f-9719-5aa7c76561a2-iptables-alerter-script\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.105414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105400 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysconfig\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cnibin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-multus\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-ovn\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38b2552-896f-498f-9719-5aa7c76561a2-host-slash\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-tuned\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovn-node-metrics-cert\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2ebff0e-2c89-4345-8577-9e296f299461-serviceca\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.105623 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-device-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgg94\" (UniqueName: \"kubernetes.io/projected/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kube-api-access-zgg94\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/25ed0359-9b20-49e1-be46-5da77b7853d3-kube-api-access-wdrkq\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-sys\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/506e21c2-5241-44f6-8c30-0581ebe68600-hosts-file\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-hostroot\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-node-log\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2ebff0e-2c89-4345-8577-9e296f299461-host\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-cnibin\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-script-lib\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.105884 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6449c7f8-c8f4-4022-8384-a85070a6493c-agent-certs\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.106297 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.105896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6449c7f8-c8f4-4022-8384-a85070a6493c-konnectivity-ca\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.146190 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.146167 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qmmzq\"" Nov 02 08:05:13.148673 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.148657 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.149123 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149104 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bgzdd\"" Nov 02 08:05:13.149217 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149160 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-c87zx\"" Nov 02 08:05:13.149291 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149275 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 08:05:13.149342 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149328 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.149394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149346 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.149394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149352 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2bt7\"" Nov 02 08:05:13.149394 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149366 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.149500 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149411 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rwld\"" Nov 02 08:05:13.149500 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149419 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 08:05:13.149500 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149420 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 08:05:13.149500 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149454 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zrnf9\"" Nov 02 08:05:13.149500 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149465 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.149696 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149519 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 08:05:13.149696 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149536 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 08:05:13.149696 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149563 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.149696 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149535 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 08:05:13.149823 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149727 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.149823 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149786 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 08:05:13.149823 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149793 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:13.149929 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.149831 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b5r9x" Nov 02 08:05:13.206844 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.206844 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-systemd-units\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-var-lib-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-etc-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206942 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-var-lib-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206934 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-etc-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-systemd\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-lib-modules\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207005 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-systemd\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.206946 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-systemd-units\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207016 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5hl\" (UniqueName: \"kubernetes.io/projected/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-kube-api-access-4h5hl\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-etc-kubernetes\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-kubelet\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207086 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-lib-modules\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r2w\" (UniqueName: \"kubernetes.io/projected/d38b2552-896f-498f-9719-5aa7c76561a2-kube-api-access-h9r2w\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207114 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-kubelet\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207114 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-etc-kubernetes\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-kubernetes\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmhs\" (UniqueName: \"kubernetes.io/projected/506e21c2-5241-44f6-8c30-0581ebe68600-kube-api-access-gsmhs\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-netns\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-conf-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207303 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-kubernetes\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-systemd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-log-socket\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207353 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-netns\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-system-cni-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.207597 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-modprobe-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207397 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-conf-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-kubelet-config\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207420 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-log-socket\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207402 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-systemd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-socket-dir-parent\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207445 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-system-cni-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-socket-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-registration-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207500 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-kubelet-config\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-modprobe-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-tmp\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-multus-certs\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207553 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-registration-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rz5\" (UniqueName: \"kubernetes.io/projected/89367e5b-7b56-4918-95f3-5ae7c2ad5293-kube-api-access-c5rz5\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207581 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-socket-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.208379 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207597 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-multus-certs\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznrq\" (UniqueName: \"kubernetes.io/projected/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-kube-api-access-tznrq\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207633 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-run\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207591 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-socket-dir-parent\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-host\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-k8s-cni-cncf-io\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207716 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-daemon-config\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207713 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207770 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-host\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207774 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-config\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207793 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-run\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207799 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-k8s-cni-cncf-io\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209156 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207776 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207826 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506e21c2-5241-44f6-8c30-0581ebe68600-tmp-dir\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-system-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468nc\" (UniqueName: \"kubernetes.io/projected/c2ebff0e-2c89-4345-8577-9e296f299461-kube-api-access-468nc\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38b2552-896f-498f-9719-5aa7c76561a2-iptables-alerter-script\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.207992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208024 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-system-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysconfig\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cnibin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-multus\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-ovn\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38b2552-896f-498f-9719-5aa7c76561a2-host-slash\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-tuned\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovn-node-metrics-cert\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.209909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2ebff0e-2c89-4345-8577-9e296f299461-serviceca\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208259 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-daemon-config\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208303 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-device-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208308 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506e21c2-5241-44f6-8c30-0581ebe68600-tmp-dir\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgg94\" (UniqueName: \"kubernetes.io/projected/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kube-api-access-zgg94\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/25ed0359-9b20-49e1-be46-5da77b7853d3-kube-api-access-wdrkq\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208374 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-ovn\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-sys\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/506e21c2-5241-44f6-8c30-0581ebe68600-hosts-file\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208404 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-config\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-hostroot\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208459 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-node-log\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208478 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208507 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysconfig\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208511 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cnibin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.210734 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2ebff0e-2c89-4345-8577-9e296f299461-host\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208548 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38b2552-896f-498f-9719-5aa7c76561a2-iptables-alerter-script\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-cnibin\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208558 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-node-log\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208615 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2ebff0e-2c89-4345-8577-9e296f299461-host\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208529 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38b2552-896f-498f-9719-5aa7c76561a2-host-slash\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208844 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-sys\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208889 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/506e21c2-5241-44f6-8c30-0581ebe68600-hosts-file\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208906 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-cnibin\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208528 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-device-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208954 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-multus\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208924 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208975 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.208982 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2ebff0e-2c89-4345-8577-9e296f299461-serviceca\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209017 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-hostroot\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.209076 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:13.211613 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.209128 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret podName:73d52e94-07b5-4a34-b8ec-74e17ad7d096 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:13.709114062 +0000 UTC m=+2.290675528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret") pod "global-pull-secret-syncer-2phd8" (UID: "73d52e94-07b5-4a34-b8ec-74e17ad7d096") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-script-lib\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209183 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6449c7f8-c8f4-4022-8384-a85070a6493c-agent-certs\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6449c7f8-c8f4-4022-8384-a85070a6493c-konnectivity-ca\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-conf\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-kubelet\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-dbus\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-bin\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cni-binary-copy\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-bin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209373 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-conf\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-netns\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209415 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-var-lib-kubelet\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209443 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-dbus\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-slash\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209485 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-slash\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-netd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212194 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209527 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-bin\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209566 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-script-lib\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209584 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-kubelet\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggvl\" (UniqueName: \"kubernetes.io/projected/ce7abe8e-c9a4-4128-a744-0a2cd073c535-kube-api-access-7ggvl\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209620 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-bin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209623 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-sys-fs\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-env-overrides\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209670 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-sys-fs\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209730 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-var-lib-kubelet\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209756 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6449c7f8-c8f4-4022-8384-a85070a6493c-konnectivity-ca\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-netns\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209826 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-netd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.209840 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.209842 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.209890 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:13.70987282 +0000 UTC m=+2.291434281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.210261 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-env-overrides\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.212651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.210292 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cni-binary-copy\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.213116 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.211618 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovn-node-metrics-cert\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.213116 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.211619 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-tuned\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.213116 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.211636 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-tmp\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.213116 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.212100 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6449c7f8-c8f4-4022-8384-a85070a6493c-agent-certs\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.264608 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.264123 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:13.264608 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.264592 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:13.264608 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.264611 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z9g8j for pod openshift-network-diagnostics/network-check-target-8d2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:13.265134 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.265110 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j podName:f68e237e-b990-4625-ba3a-a02cdb285bd8 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:13.764658486 +0000 UTC m=+2.346219953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z9g8j" (UniqueName: "kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j") pod "network-check-target-8d2bg" (UID: "f68e237e-b990-4625-ba3a-a02cdb285bd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:13.265752 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.265675 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5hl\" (UniqueName: \"kubernetes.io/projected/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-kube-api-access-4h5hl\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.265752 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.265699 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmhs\" (UniqueName: \"kubernetes.io/projected/506e21c2-5241-44f6-8c30-0581ebe68600-kube-api-access-gsmhs\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.265900 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.265869 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r2w\" (UniqueName: \"kubernetes.io/projected/d38b2552-896f-498f-9719-5aa7c76561a2-kube-api-access-h9r2w\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.265960 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.265907 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rz5\" (UniqueName: \"kubernetes.io/projected/89367e5b-7b56-4918-95f3-5ae7c2ad5293-kube-api-access-c5rz5\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.266351 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.266328 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/25ed0359-9b20-49e1-be46-5da77b7853d3-kube-api-access-wdrkq\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.266902 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.266877 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-468nc\" (UniqueName: \"kubernetes.io/projected/c2ebff0e-2c89-4345-8577-9e296f299461-kube-api-access-468nc\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.267181 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.267164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznrq\" (UniqueName: \"kubernetes.io/projected/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-kube-api-access-tznrq\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.267674 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.267656 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggvl\" (UniqueName: \"kubernetes.io/projected/ce7abe8e-c9a4-4128-a744-0a2cd073c535-kube-api-access-7ggvl\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.267836 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.267820 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgg94\" (UniqueName: \"kubernetes.io/projected/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kube-api-access-zgg94\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.378596 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.378563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:13.387382 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.387353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:13.411226 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.411191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:13.413308 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.413283 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:13.413876 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.413855 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:13.430626 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.430590 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4xwk" Nov 02 08:05:13.451503 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.451470 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:13.455974 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.455955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:13.476626 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.476599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:13.519540 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:13.519470 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104fe0f8525d94a2e03d5f4707bfa4d0.slice/crio-c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67 WatchSource:0}: Error finding container c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67: Status 404 returned error can't find the container with id c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67 Nov 02 08:05:13.519932 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:13.519903 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d28ca7a91586006c95d24bde05028b.slice/crio-57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe WatchSource:0}: Error finding container 57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe: Status 404 returned error can't find the container with id 57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe Nov 02 08:05:13.524887 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.524870 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:05:13.683263 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.683233 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:13.713688 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.713657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:13.713840 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.713726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:13.713840 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.713810 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:13.713905 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.713862 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:14.713849972 +0000 UTC m=+3.295411421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:13.713905 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.713810 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:13.713905 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.713895 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret podName:73d52e94-07b5-4a34-b8ec-74e17ad7d096 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:14.713885744 +0000 UTC m=+3.295447202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret") pod "global-pull-secret-syncer-2phd8" (UID: "73d52e94-07b5-4a34-b8ec-74e17ad7d096") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:13.814731 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.814648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:13.815264 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.814764 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:13.815264 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.814781 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:13.815264 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.814790 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z9g8j for pod openshift-network-diagnostics/network-check-target-8d2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:13.815264 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:13.814837 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j podName:f68e237e-b990-4625-ba3a-a02cdb285bd8 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:14.814825205 +0000 UTC m=+3.396386663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9g8j" (UniqueName: "kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j") pod "network-check-target-8d2bg" (UID: "f68e237e-b990-4625-ba3a-a02cdb285bd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:13.866829 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.866775 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"34d28ca7a91586006c95d24bde05028b","Type":"ContainerStarted","Data":"57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe"} Nov 02 08:05:13.867715 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:13.867692 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"104fe0f8525d94a2e03d5f4707bfa4d0","Type":"ContainerStarted","Data":"c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67"} Nov 02 08:05:14.066436 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.066355 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:14.153226 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.153167 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 08:00:12 +0000 UTC" deadline="2028-08-16 22:27:23.30039094 +0000 UTC" Nov 02 08:05:14.153226 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.153218 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15686h22m9.147176957s" Nov 02 08:05:14.173160 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.173120 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:14.260071 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.260024 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b96f0e5_7abd_41ca_ae54_7c8f35d5cb86.slice/crio-753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e WatchSource:0}: Error finding container 753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e: Status 404 returned error can't find the container with id 753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e Nov 02 08:05:14.296049 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.296023 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3835ee_72cf_4adc_a485_b9c55d8601ce.slice/crio-d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad WatchSource:0}: Error finding container d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad: Status 404 returned error can't find the container with id d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad Nov 02 08:05:14.321606 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.321581 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506e21c2_5241_44f6_8c30_0581ebe68600.slice/crio-e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b WatchSource:0}: Error finding container e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b: Status 404 returned error can't find the container with id e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b Nov 02 08:05:14.358701 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.358672 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ed0359_9b20_49e1_be46_5da77b7853d3.slice/crio-3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a WatchSource:0}: Error finding container 3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a: Status 404 returned error can't find the container with id 3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a Nov 02 08:05:14.359680 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.359658 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6449c7f8_c8f4_4022_8384_a85070a6493c.slice/crio-8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711 WatchSource:0}: Error finding container 8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711: Status 404 returned error can't find the container with id 8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711 Nov 02 08:05:14.387466 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.387440 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc3fb78_91ec_45b5_a9e2_a7f1101c8a04.slice/crio-cfac2ef7ca61f32ac1a53ab8b34725dc085117b8b2ffd94dda8f7c79c068f8dc WatchSource:0}: Error finding container cfac2ef7ca61f32ac1a53ab8b34725dc085117b8b2ffd94dda8f7c79c068f8dc: Status 404 returned error can't find the container with id cfac2ef7ca61f32ac1a53ab8b34725dc085117b8b2ffd94dda8f7c79c068f8dc Nov 02 08:05:14.419388 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.419360 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38b2552_896f_498f_9719_5aa7c76561a2.slice/crio-b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85 WatchSource:0}: Error finding container b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85: Status 404 returned error can't find the container with id b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85 Nov 02 08:05:14.501293 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:14.501265 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ebff0e_2c89_4345_8577_9e296f299461.slice/crio-aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728 WatchSource:0}: Error finding container aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728: Status 404 returned error can't find the container with id aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728 Nov 02 08:05:14.720567 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.720480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:14.720567 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.720542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:14.720902 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.720704 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:14.720902 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.720774 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:16.720755721 +0000 UTC m=+5.302317187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:14.720902 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.720824 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:14.720902 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.720896 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret podName:73d52e94-07b5-4a34-b8ec-74e17ad7d096 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:16.720877949 +0000 UTC m=+5.302439394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret") pod "global-pull-secret-syncer-2phd8" (UID: "73d52e94-07b5-4a34-b8ec-74e17ad7d096") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:14.821480 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.821438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:14.822086 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.821604 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:14.822086 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.821621 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:14.822086 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.821631 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z9g8j for pod openshift-network-diagnostics/network-check-target-8d2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:14.822086 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.821678 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j podName:f68e237e-b990-4625-ba3a-a02cdb285bd8 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:16.821665115 +0000 UTC m=+5.403226573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9g8j" (UniqueName: "kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j") pod "network-check-target-8d2bg" (UID: "f68e237e-b990-4625-ba3a-a02cdb285bd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:14.863415 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.863223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:14.863415 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.863273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:14.863415 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.863347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:14.863415 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.863364 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:14.863732 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.863457 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:14.863732 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:14.863535 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:14.871308 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.871273 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" event={"ID":"ae3835ee-72cf-4adc-a485-b9c55d8601ce","Type":"ContainerStarted","Data":"d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad"} Nov 02 08:05:14.872746 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.872660 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e"} Nov 02 08:05:14.874921 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.874513 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4xwk" event={"ID":"89367e5b-7b56-4918-95f3-5ae7c2ad5293","Type":"ContainerStarted","Data":"2058ac1dfa1098d746ac969bcaf0d27ef177bce954adf7195debaf39d8c7613c"} Nov 02 08:05:14.875899 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.875875 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hlm8n" event={"ID":"6449c7f8-c8f4-4022-8384-a85070a6493c","Type":"ContainerStarted","Data":"8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711"} Nov 02 08:05:14.877208 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.877186 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m29jr" event={"ID":"506e21c2-5241-44f6-8c30-0581ebe68600","Type":"ContainerStarted","Data":"e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b"} Nov 02 08:05:14.878564 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.878538 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s52jj" event={"ID":"c2ebff0e-2c89-4345-8577-9e296f299461","Type":"ContainerStarted","Data":"aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728"} Nov 02 08:05:14.880775 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.880751 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vc95q" event={"ID":"d38b2552-896f-498f-9719-5aa7c76561a2","Type":"ContainerStarted","Data":"b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85"} Nov 02 08:05:14.882271 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.882238 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nldh5" event={"ID":"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04","Type":"ContainerStarted","Data":"cfac2ef7ca61f32ac1a53ab8b34725dc085117b8b2ffd94dda8f7c79c068f8dc"} Nov 02 08:05:14.883503 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:14.883477 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerStarted","Data":"3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a"} Nov 02 08:05:15.154332 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:15.154222 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 08:00:12 +0000 UTC" deadline="2028-07-09 03:47:25.034902372 +0000 UTC" Nov 02 08:05:15.154332 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:15.154279 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14755h42m9.880628s" Nov 02 08:05:16.741384 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.741104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:16.741954 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.741423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:16.741954 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.741588 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:16.741954 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.741657 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:20.741635746 +0000 UTC m=+9.323197211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:16.741954 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.741725 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:16.741954 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.741761 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret podName:73d52e94-07b5-4a34-b8ec-74e17ad7d096 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:20.741748434 +0000 UTC m=+9.323309893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret") pod "global-pull-secret-syncer-2phd8" (UID: "73d52e94-07b5-4a34-b8ec-74e17ad7d096") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:16.842504 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.842382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:16.842734 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.842611 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:16.842734 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.842632 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:16.842734 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.842642 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z9g8j for pod openshift-network-diagnostics/network-check-target-8d2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:16.842734 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.842684 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j podName:f68e237e-b990-4625-ba3a-a02cdb285bd8 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:20.842673292 +0000 UTC m=+9.424234737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9g8j" (UniqueName: "kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j") pod "network-check-target-8d2bg" (UID: "f68e237e-b990-4625-ba3a-a02cdb285bd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:16.863669 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.863639 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:16.863849 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.863774 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:16.863849 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.863835 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:16.863955 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.863914 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:16.864010 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.863956 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:16.864071 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:16.864040 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:16.907248 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.907206 2582 generic.go:356] "Generic (PLEG): container finished" podID="104fe0f8525d94a2e03d5f4707bfa4d0" containerID="cc2b1b24f28cb996e2d8395f9b1346cd2aa2d6ab2f05cb199f916e1ef0162f75" exitCode=0 Nov 02 08:05:16.907420 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:16.907305 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"104fe0f8525d94a2e03d5f4707bfa4d0","Type":"ContainerDied","Data":"cc2b1b24f28cb996e2d8395f9b1346cd2aa2d6ab2f05cb199f916e1ef0162f75"} Nov 02 08:05:17.924891 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:17.924256 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"34d28ca7a91586006c95d24bde05028b","Type":"ContainerStarted","Data":"42310aa681edcd3e7d39ccab8ec10589e393c41af35d14666a4779fcdbdae45e"} Nov 02 08:05:17.930601 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:17.930110 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"104fe0f8525d94a2e03d5f4707bfa4d0","Type":"ContainerStarted","Data":"dc61793d9e3592958db8157fc033375f7086a294b52861a33c1e8a3a80a866e2"} Nov 02 08:05:18.150989 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:18.150929 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" podStartSLOduration=6.150908317 podStartE2EDuration="6.150908317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:18.038057274 +0000 UTC m=+6.619618741" watchObservedRunningTime="2026-11-02 08:05:18.150908317 +0000 UTC m=+6.732469787" Nov 02 08:05:18.863819 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:18.863782 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:18.864043 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:18.864023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:18.864153 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:18.864121 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:18.864224 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:18.864215 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:18.864320 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:18.864290 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:18.864381 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:18.863930 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:18.939827 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:18.939792 2582 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="f72c70d8524fb09ca9e26ebe27bbcbe1d9bd656fb7aeefe594196ee11ae7a1e0" exitCode=0 Nov 02 08:05:18.940784 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:18.940756 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerDied","Data":"f72c70d8524fb09ca9e26ebe27bbcbe1d9bd656fb7aeefe594196ee11ae7a1e0"} Nov 02 08:05:19.151157 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:19.151086 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" podStartSLOduration=7.151064463 podStartE2EDuration="7.151064463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:18.15158609 +0000 UTC m=+6.733147558" watchObservedRunningTime="2026-11-02 08:05:19.151064463 +0000 UTC m=+7.732625931" Nov 02 08:05:19.946478 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:19.946424 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hlm8n" event={"ID":"6449c7f8-c8f4-4022-8384-a85070a6493c","Type":"ContainerStarted","Data":"83a20841d2cea1e9d176950e755d518031b69510c777ff7df395821a16adc501"} Nov 02 08:05:20.777252 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.777210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:20.777480 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.777273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:20.777480 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.777435 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:20.777632 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.777500 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:28.777481186 +0000 UTC m=+17.359042646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:20.777632 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.777567 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:20.777632 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.777599 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret podName:73d52e94-07b5-4a34-b8ec-74e17ad7d096 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:28.777588861 +0000 UTC m=+17.359150322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret") pod "global-pull-secret-syncer-2phd8" (UID: "73d52e94-07b5-4a34-b8ec-74e17ad7d096") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:20.863781 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.863739 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:20.863781 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.863784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:20.864013 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.863745 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:20.864013 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.863922 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:20.864121 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.864013 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:20.864121 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.864102 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:20.878135 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.878094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:20.878320 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.878255 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:20.878320 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.878280 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:20.878320 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.878294 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z9g8j for pod openshift-network-diagnostics/network-check-target-8d2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:20.878480 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:20.878362 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j podName:f68e237e-b990-4625-ba3a-a02cdb285bd8 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:28.878342648 +0000 UTC m=+17.459904129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9g8j" (UniqueName: "kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j") pod "network-check-target-8d2bg" (UID: "f68e237e-b990-4625-ba3a-a02cdb285bd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:20.963347 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:20.963206 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s52jj" event={"ID":"c2ebff0e-2c89-4345-8577-9e296f299461","Type":"ContainerStarted","Data":"f3c763610d68c6bba194c9920cd8ffa905339a8530f21a7e03f608903454d5ee"} Nov 02 08:05:21.075462 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:21.075398 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hlm8n" podStartSLOduration=4.289680824 podStartE2EDuration="9.07537511s" totalImagesPullingTime="4.785694286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:20.05973391 +0000 UTC m=+8.641295381" watchObservedRunningTime="2026-11-02 08:05:21.07537511 +0000 UTC m=+9.656936578" Nov 02 08:05:21.968279 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:21.968192 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" event={"ID":"ae3835ee-72cf-4adc-a485-b9c55d8601ce","Type":"ContainerStarted","Data":"2bfe73b9bb4e1955ad237b7c3fc9bb579f55be170882bf35cb197bd3f08572f3"} Nov 02 08:05:22.175695 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.175664 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:22.176570 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.176523 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:22.288757 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.288652 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s52jj" podStartSLOduration=4.49398289 podStartE2EDuration="10.288630305s" totalImagesPullingTime="5.794647415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:21.076601242 +0000 UTC m=+9.658162710" watchObservedRunningTime="2026-11-02 08:05:22.288630305 +0000 UTC m=+10.870191775" Nov 02 08:05:22.863540 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.863454 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:22.863711 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:22.863599 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:22.863780 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.863710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:22.863838 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:22.863803 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:22.863838 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.863834 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:22.863938 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:22.863904 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:22.975932 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.975853 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4xwk" event={"ID":"89367e5b-7b56-4918-95f3-5ae7c2ad5293","Type":"ContainerStarted","Data":"5a6986013da2c9e46df2bc194e9623e65258900af20e207d36fcb173a5838b95"} Nov 02 08:05:22.980552 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:22.980484 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m29jr" event={"ID":"506e21c2-5241-44f6-8c30-0581ebe68600","Type":"ContainerStarted","Data":"150f7dca38bb35b0671ebe33cb3894204db22098bc3acd34621ec28b1122ad47"} Nov 02 08:05:23.145407 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.145379 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 08:05:23.204456 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.204405 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g4xwk" podStartSLOduration=2.993997783 podStartE2EDuration="11.204391014s" totalImagesPullingTime="8.210393231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:23.091345059 +0000 UTC m=+11.672906530" watchObservedRunningTime="2026-11-02 08:05:23.204391014 +0000 UTC m=+11.785952477" Nov 02 08:05:23.868492 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.868155 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T08:05:23.145401012Z","UUID":"ed524e67-19c9-45b8-b4a1-177ea70c3c83","Handler":null,"Name":"","Endpoint":""} Nov 02 08:05:23.871364 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.871338 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 08:05:23.871364 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.871369 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 08:05:23.984774 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.984667 2582 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="69889d2c64483954688b1352216bbbf6fbf22ac5078f6f8eac36056d4a031751" exitCode=0 Nov 02 08:05:23.984774 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.984739 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerDied","Data":"69889d2c64483954688b1352216bbbf6fbf22ac5078f6f8eac36056d4a031751"} Nov 02 08:05:23.988382 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:23.988358 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" event={"ID":"ae3835ee-72cf-4adc-a485-b9c55d8601ce","Type":"ContainerStarted","Data":"642ad7f16638d0bae208952df7dc36a8a0664e1c28970e56c4d6dc971143150d"} Nov 02 08:05:24.102904 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:24.102855 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-m29jr" podStartSLOduration=3.69978833 podStartE2EDuration="12.102839945s" totalImagesPullingTime="8.403051615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:23.205174359 +0000 UTC m=+11.786735827" watchObservedRunningTime="2026-11-02 08:05:24.102839945 +0000 UTC m=+12.684401412" Nov 02 08:05:24.863651 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:24.863558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:24.863807 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:24.863703 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:24.863807 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:24.863777 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:24.863921 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:24.863841 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:24.863921 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:24.863893 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:24.864034 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:24.863953 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:24.994445 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:24.993268 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vc95q" event={"ID":"d38b2552-896f-498f-9719-5aa7c76561a2","Type":"ContainerStarted","Data":"77c356d4970c0782fff4aa501413c3323dd4bb2a8ed15bc70f4276b61f2527f5"} Nov 02 08:05:24.996243 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:24.995784 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" event={"ID":"ae3835ee-72cf-4adc-a485-b9c55d8601ce","Type":"ContainerStarted","Data":"fc7f2dd44ee19a9d1d9604bfd991651631ad48f4c5047805fae73cb812e95178"} Nov 02 08:05:25.105008 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:25.104947 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vc95q" podStartSLOduration=4.49704516 podStartE2EDuration="13.10493051s" totalImagesPullingTime="8.60788535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:25.104829242 +0000 UTC m=+13.686390712" watchObservedRunningTime="2026-11-02 08:05:25.10493051 +0000 UTC m=+13.686491979" Nov 02 08:05:25.999892 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:25.999853 2582 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="2642834de699f4969375c606d44131b3ed330b56cb112b510258b75441e38641" exitCode=0 Nov 02 08:05:26.000456 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:25.999945 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerDied","Data":"2642834de699f4969375c606d44131b3ed330b56cb112b510258b75441e38641"} Nov 02 08:05:26.002297 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:26.002236 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nldh5" event={"ID":"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04","Type":"ContainerStarted","Data":"61d8e4dbbd541a3d0b758542883fa016454c70087ef1057a2759794000f8addc"} Nov 02 08:05:26.219911 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:26.219858 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" podStartSLOduration=4.123358747 podStartE2EDuration="14.219844135s" totalImagesPullingTime="10.096485388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:25.219068901 +0000 UTC m=+13.800630364" watchObservedRunningTime="2026-11-02 08:05:26.219844135 +0000 UTC m=+14.801405602" Nov 02 08:05:26.334570 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:26.334462 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nldh5" podStartSLOduration=3.768611342 podStartE2EDuration="14.334448175s" totalImagesPullingTime="10.565836833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:26.333788478 +0000 UTC m=+14.915349945" watchObservedRunningTime="2026-11-02 08:05:26.334448175 +0000 UTC m=+14.916009641" Nov 02 08:05:26.864165 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:26.864093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:26.864165 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:26.864115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:26.864393 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:26.864247 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:26.864393 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:26.864300 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:26.864501 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:26.864409 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:26.864704 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:26.864675 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:28.009059 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.009024 2582 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="570abe045677af91fe6ef8075531c888f4bdac100d4c21735fd0c0d8d925f39b" exitCode=0 Nov 02 08:05:28.009632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.009088 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerDied","Data":"570abe045677af91fe6ef8075531c888f4bdac100d4c21735fd0c0d8d925f39b"} Nov 02 08:05:28.843707 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.843578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:28.843812 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.843709 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:28.843812 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.843767 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:44.843750198 +0000 UTC m=+33.425311646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:28.843812 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.843787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:28.843929 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.843855 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:28.843929 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.843883 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret podName:73d52e94-07b5-4a34-b8ec-74e17ad7d096 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:44.843875873 +0000 UTC m=+33.425437319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret") pod "global-pull-secret-syncer-2phd8" (UID: "73d52e94-07b5-4a34-b8ec-74e17ad7d096") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:28.863062 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.863040 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:28.863181 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.863045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:28.863181 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.863152 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:28.863181 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.863171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:28.863327 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.863260 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:28.863376 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.863346 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:28.945043 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:28.945013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:28.945181 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.945163 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:28.945251 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.945183 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:28.945251 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.945196 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z9g8j for pod openshift-network-diagnostics/network-check-target-8d2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:28.945251 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:28.945249 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j podName:f68e237e-b990-4625-ba3a-a02cdb285bd8 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:44.945234971 +0000 UTC m=+33.526796437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9g8j" (UniqueName: "kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j") pod "network-check-target-8d2bg" (UID: "f68e237e-b990-4625-ba3a-a02cdb285bd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:29.013802 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:29.013773 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"c43169a4708d18fcee3b3f4069cef034f232426ca1aa7008e820177f52a20177"} Nov 02 08:05:29.014155 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:29.013855 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"ec76b5b736b38fdb46eb498736b8646d71d2348f9e5b3300b875a2943913d495"} Nov 02 08:05:29.014155 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:29.013886 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"45dc3d69d6748a90da676c353a5a8a406e51fb4c109afc9ea5b0352fbd6c6cf5"} Nov 02 08:05:30.020040 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:30.019855 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"16bfd11ff2f54ec2a6c69fef05342871f2dfb02fb01a80fab232ef55636c61e5"} Nov 02 08:05:30.020440 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:30.020078 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"f4809b79e7fb9723a77dba0478265c5a35a1cacae2e8833ac447ee4bff11e6bf"} Nov 02 08:05:30.020440 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:30.020114 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"5f7d3f73a4b997c0edebe58c5e0a859e910f691bfa0b16c692b7c327e084884c"} Nov 02 08:05:30.863870 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:30.863816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:30.863870 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:30.863850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:30.863870 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:30.863854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:30.864101 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:30.863951 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:30.864101 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:30.864023 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:30.864101 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:30.864071 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:31.027515 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:31.027480 2582 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="86fe0a1b52e28d2658a92c7a99bad14a3e8e85a438297f9fdb367514be722599" exitCode=0 Nov 02 08:05:31.027985 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:31.027520 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerDied","Data":"86fe0a1b52e28d2658a92c7a99bad14a3e8e85a438297f9fdb367514be722599"} Nov 02 08:05:32.032582 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:32.032542 2582 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="8f5d8fdadbb1f2431736e8f6356cd8a45809cc942c43569276ea607738736a36" exitCode=0 Nov 02 08:05:32.033119 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:32.032613 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerDied","Data":"8f5d8fdadbb1f2431736e8f6356cd8a45809cc942c43569276ea607738736a36"} Nov 02 08:05:32.036014 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:32.035989 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"86bcece153054aaed0e57efdc70a1b57859e5dc72908bcb2aa2715a5f8544c22"} Nov 02 08:05:32.864110 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:32.864068 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:32.864292 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:32.864075 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:32.864292 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:32.864082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:32.864292 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:32.864214 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:32.864384 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:32.864301 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:32.864417 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:32.864379 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:33.040958 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:33.040922 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phvgd" event={"ID":"25ed0359-9b20-49e1-be46-5da77b7853d3","Type":"ContainerStarted","Data":"a52b947b89685c0661fdc34f4254a8bfd86884e9eb4f566ae03dd72b78c146da"} Nov 02 08:05:33.161886 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:33.161792 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phvgd" podStartSLOduration=8.514202139 podStartE2EDuration="21.16178012s" totalImagesPullingTime="12.647577981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:33.161185352 +0000 UTC m=+21.742746816" watchObservedRunningTime="2026-11-02 08:05:33.16178012 +0000 UTC m=+21.743341587" Nov 02 08:05:34.046435 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:34.046246 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" event={"ID":"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86","Type":"ContainerStarted","Data":"b2bcca2149721b4d75bf8ecde6dff79e52bd644ca30723a15cf611f6a25749ce"} Nov 02 08:05:34.320322 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:34.320230 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" podStartSLOduration=7.760558797 podStartE2EDuration="22.32021565s" totalImagesPullingTime="14.559656853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:34.269439654 +0000 UTC m=+22.851001122" watchObservedRunningTime="2026-11-02 08:05:34.32021565 +0000 UTC m=+22.901777490" Nov 02 08:05:34.863581 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:34.863546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:34.863772 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:34.863548 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:34.863772 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:34.863675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:34.863772 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:34.863750 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:34.863880 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:34.863656 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:34.863880 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:34.863835 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:35.048917 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:35.048889 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:05:35.049443 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:35.049387 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:35.049505 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:35.049467 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:35.065267 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:35.065239 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:35.065433 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:35.065363 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:36.050045 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.050011 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2phd8"] Nov 02 08:05:36.051250 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.050160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:36.051250 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:36.050268 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:36.051637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.051607 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:05:36.053855 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.053831 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8d2bg"] Nov 02 08:05:36.054010 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.053937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:36.054049 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:36.054032 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:36.054411 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.054390 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-trgc8"] Nov 02 08:05:36.054492 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.054481 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:36.054584 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:36.054567 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:36.871675 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:36.871636 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:37.863804 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:37.863725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:37.863804 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:37.863756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:37.864205 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:37.863725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:37.864205 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:37.863843 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:37.864205 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:37.863908 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:37.864205 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:37.863957 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:38.073524 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:38.073485 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" podUID="0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 08:05:39.863168 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:39.863124 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:39.863626 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:39.863130 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:39.863626 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:39.863244 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d2bg" podUID="f68e237e-b990-4625-ba3a-a02cdb285bd8" Nov 02 08:05:39.863626 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:39.863262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:39.863626 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:39.863325 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trgc8" podUID="ce7abe8e-c9a4-4128-a744-0a2cd073c535" Nov 02 08:05:39.863626 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:39.863456 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2phd8" podUID="73d52e94-07b5-4a34-b8ec-74e17ad7d096" Nov 02 08:05:40.314471 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:40.314381 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:40.314634 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:40.314520 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:05:40.315023 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:40.314998 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:41.668812 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.668633 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeReady" Nov 02 08:05:41.669205 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.668942 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 08:05:41.782965 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.782934 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-86sd2"] Nov 02 08:05:41.796559 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.796532 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qmp75"] Nov 02 08:05:41.796810 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.796667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:41.805301 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.805279 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cbw8w"] Nov 02 08:05:41.805421 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.805387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:41.814855 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.814467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:41.834088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.834060 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-595bc5b6bc-js2wl"] Nov 02 08:05:41.841634 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.841613 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fr2k5"] Nov 02 08:05:41.841740 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.841728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.852099 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.852073 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f"] Nov 02 08:05:41.852246 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.852219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.864070 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.864044 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx"] Nov 02 08:05:41.864232 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.864166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:41.864392 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.864360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:41.864761 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.864743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:41.864973 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.864953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:41.878225 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.878202 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59f6c46786-fzsst"] Nov 02 08:05:41.878356 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.878338 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:41.884492 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.884465 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 08:05:41.884492 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.884480 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 08:05:41.892973 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.892953 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd"] Nov 02 08:05:41.893174 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.893153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:41.906042 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.906023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:41.916756 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.916732 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8"] Nov 02 08:05:41.929822 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.929794 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt"] Nov 02 08:05:41.929944 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.929923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:41.934326 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-snapshots\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.934497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934340 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-serving-cert\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.934497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfx5\" (UniqueName: \"kubernetes.io/projected/1d14a7ee-7798-4273-9e07-68f3ad95b363-kube-api-access-7jfx5\") pod \"kas-connection-checker-685895c7c9-cbw8w\" (UID: \"1d14a7ee-7798-4273-9e07-68f3ad95b363\") " pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:41.934497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934399 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 08:05:41.934497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.934497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934507 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934560 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934566 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w8vvg\"" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05387920-8354-45bf-85e3-513462f484bf-serving-cert\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6v6\" (UniqueName: \"kubernetes.io/projected/05387920-8354-45bf-85e3-513462f484bf-kube-api-access-dx6v6\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:41.934732 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-service-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.934946 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05387920-8354-45bf-85e3-513462f484bf-config\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:41.934946 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:41.934946 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.934946 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.934946 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934900 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:41.935112 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.934955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.935112 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.935202 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.935252 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-tmp\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.935252 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z2w4\" (UniqueName: \"kubernetes.io/projected/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-kube-api-access-8z2w4\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:41.935343 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:41.935539 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qvk\" (UniqueName: \"kubernetes.io/projected/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd-kube-api-access-p6qvk\") pod \"kas-connection-checker-685895c7c9-qmp75\" (UID: \"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd\") " pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:41.935539 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wkgx\" (UniqueName: \"kubernetes.io/projected/163e10f5-9315-4873-82b8-f830ff3135ac-kube-api-access-9wkgx\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:41.935539 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.935439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xth\" (UniqueName: \"kubernetes.io/projected/e38a56d4-3eca-4c4d-b7f9-fbd80403e692-kube-api-access-74xth\") pod \"kas-connection-checker-685895c7c9-86sd2\" (UID: \"e38a56d4-3eca-4c4d-b7f9-fbd80403e692\") " pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:41.942098 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.942079 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tspm9"] Nov 02 08:05:41.942214 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.942196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:41.954607 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.954588 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wtksb"] Nov 02 08:05:41.954706 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.954593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:41.954952 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.954931 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:41.955045 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.954931 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:41.955045 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955031 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 08:05:41.955176 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955086 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:41.955242 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955197 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 08:05:41.955242 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955215 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 08:05:41.955242 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955218 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 08:05:41.955359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955248 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 08:05:41.955429 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955361 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 08:05:41.955874 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955852 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 08:05:41.955950 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.955914 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:41.964774 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.964752 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d5df879b9-lxqd6"] Nov 02 08:05:41.964869 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.964855 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:41.976998 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.976978 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk"] Nov 02 08:05:41.977116 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.977100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:41.983300 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.983279 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q49jt"] Nov 02 08:05:41.983405 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.983389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:41.991261 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.991244 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-svhfk"] Nov 02 08:05:41.991359 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:41.991346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.001414 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001397 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-86sd2"] Nov 02 08:05:42.001493 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001422 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cbw8w"] Nov 02 08:05:42.001493 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001438 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f"] Nov 02 08:05:42.001493 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001462 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-595bc5b6bc-js2wl"] Nov 02 08:05:42.001493 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001479 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx"] Nov 02 08:05:42.001493 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001495 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fr2k5"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001508 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001536 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001550 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wtksb"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001562 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tspm9"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001574 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q49jt"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001591 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d5df879b9-lxqd6"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001604 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-svhfk"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001615 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qmp75"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001627 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt"] Nov 02 08:05:42.001638 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001640 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk"] Nov 02 08:05:42.001963 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.001963 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.001652 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f6c46786-fzsst"] Nov 02 08:05:42.007021 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.006994 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 08:05:42.007440 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.007360 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:42.007554 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.007535 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:42.007744 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.007724 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:42.008048 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.008031 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:42.010616 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.009892 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 08:05:42.010616 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.010256 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 08:05:42.010616 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.010316 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 08:05:42.010817 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.010777 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 08:05:42.010817 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.010803 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 08:05:42.010919 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.010903 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 08:05:42.011175 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011159 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 08:05:42.011313 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011295 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4v7vr\"" Nov 02 08:05:42.011864 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011846 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 08:05:42.011967 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011872 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nct\"" Nov 02 08:05:42.011967 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011884 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 08:05:42.011967 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011896 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 08:05:42.012159 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011907 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghx8m\"" Nov 02 08:05:42.012159 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011930 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 08:05:42.012159 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.011965 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 08:05:42.012159 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.012016 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 08:05:42.036240 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.036380 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.036380 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrkz\" (UniqueName: \"kubernetes.io/projected/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-kube-api-access-bbrkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.036380 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.036522 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmkz\" (UniqueName: \"kubernetes.io/projected/438dc60b-6968-41c4-b974-cff36e50ab18-kube-api-access-ndmkz\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.036522 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.036614 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkgx\" (UniqueName: \"kubernetes.io/projected/163e10f5-9315-4873-82b8-f830ff3135ac-kube-api-access-9wkgx\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:42.036614 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-tmp\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.036614 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xth\" (UniqueName: \"kubernetes.io/projected/e38a56d4-3eca-4c4d-b7f9-fbd80403e692-kube-api-access-74xth\") pod \"kas-connection-checker-685895c7c9-86sd2\" (UID: \"e38a56d4-3eca-4c4d-b7f9-fbd80403e692\") " pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:42.036744 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z2w4\" (UniqueName: \"kubernetes.io/projected/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-kube-api-access-8z2w4\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.036744 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlxc\" (UniqueName: \"kubernetes.io/projected/6045c48b-d8e9-4c92-8201-3ccc48a38a91-kube-api-access-5tlxc\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.036744 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/611a6955-050e-4d16-b301-aa943a2f980e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.036744 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.036744 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:42.036979 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfx5\" (UniqueName: \"kubernetes.io/projected/1d14a7ee-7798-4273-9e07-68f3ad95b363-kube-api-access-7jfx5\") pod \"kas-connection-checker-685895c7c9-cbw8w\" (UID: \"1d14a7ee-7798-4273-9e07-68f3ad95b363\") " pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:42.036979 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.036979 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.036916 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-tmp\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.037112 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.037112 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.037112 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05387920-8354-45bf-85e3-513462f484bf-serving-cert\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.037112 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6v6\" (UniqueName: \"kubernetes.io/projected/05387920-8354-45bf-85e3-513462f484bf-kube-api-access-dx6v6\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-trusted-ca\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.037136 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-stats-auth\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.037171 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05387920-8354-45bf-85e3-513462f484bf-config\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.037221 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.537203026 +0000 UTC m=+31.118764473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jp4p\" (UniqueName: \"kubernetes.io/projected/611a6955-050e-4d16-b301-aa943a2f980e-kube-api-access-8jp4p\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.037331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-default-certificate\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vt4f\" (UniqueName: \"kubernetes.io/projected/f86f89f2-60bc-44f1-b885-e33cc92bcd9f-kube-api-access-8vt4f\") pod \"volume-data-source-validator-58c9f5f46d-d6dk8\" (UID: \"f86f89f2-60bc-44f1-b885-e33cc92bcd9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438dc60b-6968-41c4-b974-cff36e50ab18-serving-cert\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qvk\" (UniqueName: \"kubernetes.io/projected/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd-kube-api-access-p6qvk\") pod \"kas-connection-checker-685895c7c9-qmp75\" (UID: \"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd\") " pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037747 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05387920-8354-45bf-85e3-513462f484bf-config\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.037806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.037987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-snapshots\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-serving-cert\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-config\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038082 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038187 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-service-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.038315 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.038267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.040876 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.041019 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041039 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-snapshots\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041063 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05387920-8354-45bf-85e3-513462f484bf-serving-cert\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.041082 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls podName:163e10f5-9315-4873-82b8-f830ff3135ac nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.541065326 +0000 UTC m=+31.122626786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2n4f" (UID: "163e10f5-9315-4873-82b8-f830ff3135ac") : secret "samples-operator-tls" not found Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041081 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041384 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041578 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041604 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-service-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.041632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.042224 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.041783 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.043220 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.043195 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-serving-cert\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.043546 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.043524 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.056054 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056023 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 08:05:42.056300 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056218 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 08:05:42.056300 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056218 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056326 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056352 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056367 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056371 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056409 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056424 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:42.056563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.056468 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 08:05:42.057311 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.057291 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 08:05:42.104888 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.104836 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 08:05:42.105191 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105156 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 08:05:42.105289 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105194 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 08:05:42.105289 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105272 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 08:05:42.105372 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105314 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 08:05:42.105372 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105346 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4gc2j\"" Nov 02 08:05:42.105524 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105469 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9ztjt\"" Nov 02 08:05:42.105565 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105535 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 08:05:42.105643 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.105632 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t9rxp\"" Nov 02 08:05:42.109179 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.109154 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.116373 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.116349 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.118345 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.118312 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6v6\" (UniqueName: \"kubernetes.io/projected/05387920-8354-45bf-85e3-513462f484bf-kube-api-access-dx6v6\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.118439 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.118328 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qvk\" (UniqueName: \"kubernetes.io/projected/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd-kube-api-access-p6qvk\") pod \"kas-connection-checker-685895c7c9-qmp75\" (UID: \"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd\") " pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:42.118516 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.118503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkgx\" (UniqueName: \"kubernetes.io/projected/163e10f5-9315-4873-82b8-f830ff3135ac-kube-api-access-9wkgx\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:42.118566 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.118551 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfx5\" (UniqueName: \"kubernetes.io/projected/1d14a7ee-7798-4273-9e07-68f3ad95b363-kube-api-access-7jfx5\") pod \"kas-connection-checker-685895c7c9-cbw8w\" (UID: \"1d14a7ee-7798-4273-9e07-68f3ad95b363\") " pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:42.118687 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.118666 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xth\" (UniqueName: \"kubernetes.io/projected/e38a56d4-3eca-4c4d-b7f9-fbd80403e692-kube-api-access-74xth\") pod \"kas-connection-checker-685895c7c9-86sd2\" (UID: \"e38a56d4-3eca-4c4d-b7f9-fbd80403e692\") " pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:42.119898 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.119881 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z2w4\" (UniqueName: \"kubernetes.io/projected/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-kube-api-access-8z2w4\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.129576 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.129559 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:42.136703 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.136687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:42.138967 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.138951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.139008 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.138976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.139008 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.138995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrkz\" (UniqueName: \"kubernetes.io/projected/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-kube-api-access-bbrkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.139111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.139111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmkz\" (UniqueName: \"kubernetes.io/projected/438dc60b-6968-41c4-b974-cff36e50ab18-kube-api-access-ndmkz\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.139111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlxc\" (UniqueName: \"kubernetes.io/projected/6045c48b-d8e9-4c92-8201-3ccc48a38a91-kube-api-access-5tlxc\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.139111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139075 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqzl\" (UniqueName: \"kubernetes.io/projected/62e8f7ea-8d81-4b80-a527-f6e228885382-kube-api-access-hzqzl\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.139111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de009ad0-3b12-421f-a8bf-edfc92fd6b63-config-volume\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.139111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de009ad0-3b12-421f-a8bf-edfc92fd6b63-tmp-dir\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.139114 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/611a6955-050e-4d16-b301-aa943a2f980e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.139201 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.639179109 +0000 UTC m=+31.220740576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.139306 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.139345 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert podName:d6491032-ad9f-4d34-98b8-6fb3f03e4cd9 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.639332499 +0000 UTC m=+31.220893961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wtksb" (UID: "d6491032-ad9f-4d34-98b8-6fb3f03e4cd9") : secret "networking-console-plugin-cert" not found Nov 02 08:05:42.139549 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-trusted-ca\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-stats-auth\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e8f7ea-8d81-4b80-a527-f6e228885382-cert\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4qzn\" (UniqueName: \"kubernetes.io/projected/f9db17c4-a168-4dfb-a89a-88933fd6413b-kube-api-access-b4qzn\") pod \"network-check-source-6f78d59f99-2ngtk\" (UID: \"f9db17c4-a168-4dfb-a89a-88933fd6413b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139652 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139683 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jp4p\" (UniqueName: \"kubernetes.io/projected/611a6955-050e-4d16-b301-aa943a2f980e-kube-api-access-8jp4p\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.140002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.139986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-default-certificate\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt4f\" (UniqueName: \"kubernetes.io/projected/f86f89f2-60bc-44f1-b885-e33cc92bcd9f-kube-api-access-8vt4f\") pod \"volume-data-source-validator-58c9f5f46d-d6dk8\" (UID: \"f86f89f2-60bc-44f1-b885-e33cc92bcd9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140054 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438dc60b-6968-41c4-b974-cff36e50ab18-serving-cert\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140096 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/611a6955-050e-4d16-b301-aa943a2f980e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpn4d\" (UniqueName: \"kubernetes.io/projected/de009ad0-3b12-421f-a8bf-edfc92fd6b63-kube-api-access-kpn4d\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-config\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.140408 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140494 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-trusted-ca\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.140515 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.140564 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls podName:611a6955-050e-4d16-b301-aa943a2f980e nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.640549139 +0000 UTC m=+31.222110598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-srmdt" (UID: "611a6955-050e-4d16-b301-aa943a2f980e") : secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.140601 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.140706 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.140721 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.140767 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.640752014 +0000 UTC m=+31.222313479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:42.141036 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.140843 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.640830241 +0000 UTC m=+31.222391732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : configmap references non-existent config key: service-ca.crt Nov 02 08:05:42.141621 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.141590 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:42.141825 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.141792 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.142228 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.142199 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-config\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.142331 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.142207 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.142397 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.142372 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.142457 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.142396 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-stats-auth\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.142568 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.142551 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-default-certificate\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.142858 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.142840 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438dc60b-6968-41c4-b974-cff36e50ab18-serving-cert\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.143126 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.143112 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.178644 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.178583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:42.198564 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.198530 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlxc\" (UniqueName: \"kubernetes.io/projected/6045c48b-d8e9-4c92-8201-3ccc48a38a91-kube-api-access-5tlxc\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.200563 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.200534 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmkz\" (UniqueName: \"kubernetes.io/projected/438dc60b-6968-41c4-b974-cff36e50ab18-kube-api-access-ndmkz\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.202191 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.201167 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jp4p\" (UniqueName: \"kubernetes.io/projected/611a6955-050e-4d16-b301-aa943a2f980e-kube-api-access-8jp4p\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.202191 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.201552 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrkz\" (UniqueName: \"kubernetes.io/projected/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-kube-api-access-bbrkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.202191 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.202021 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.202191 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.202122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.202365 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.202228 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt4f\" (UniqueName: \"kubernetes.io/projected/f86f89f2-60bc-44f1-b885-e33cc92bcd9f-kube-api-access-8vt4f\") pod \"volume-data-source-validator-58c9f5f46d-d6dk8\" (UID: \"f86f89f2-60bc-44f1-b885-e33cc92bcd9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.241804 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqzl\" (UniqueName: \"kubernetes.io/projected/62e8f7ea-8d81-4b80-a527-f6e228885382-kube-api-access-hzqzl\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.241845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de009ad0-3b12-421f-a8bf-edfc92fd6b63-config-volume\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.241872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de009ad0-3b12-421f-a8bf-edfc92fd6b63-tmp-dir\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.241963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e8f7ea-8d81-4b80-a527-f6e228885382-cert\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.241989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qzn\" (UniqueName: \"kubernetes.io/projected/f9db17c4-a168-4dfb-a89a-88933fd6413b-kube-api-access-b4qzn\") pod \"network-check-source-6f78d59f99-2ngtk\" (UID: \"f9db17c4-a168-4dfb-a89a-88933fd6413b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.242049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.242099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpn4d\" (UniqueName: \"kubernetes.io/projected/de009ad0-3b12-421f-a8bf-edfc92fd6b63-kube-api-access-kpn4d\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.242457 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.242536 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls podName:de009ad0-3b12-421f-a8bf-edfc92fd6b63 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:42.742515357 +0000 UTC m=+31.324076802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls") pod "dns-default-q49jt" (UID: "de009ad0-3b12-421f-a8bf-edfc92fd6b63") : secret "dns-default-metrics-tls" not found Nov 02 08:05:42.246012 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.242834 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de009ad0-3b12-421f-a8bf-edfc92fd6b63-tmp-dir\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.247842 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.247489 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e8f7ea-8d81-4b80-a527-f6e228885382-cert\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.248484 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.248455 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:42.248683 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.248625 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de009ad0-3b12-421f-a8bf-edfc92fd6b63-config-volume\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.282400 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.282185 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:42.286459 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.286435 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:42.299908 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.299884 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:42.305311 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.304990 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqzl\" (UniqueName: \"kubernetes.io/projected/62e8f7ea-8d81-4b80-a527-f6e228885382-kube-api-access-hzqzl\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.305311 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.305265 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpn4d\" (UniqueName: \"kubernetes.io/projected/de009ad0-3b12-421f-a8bf-edfc92fd6b63-kube-api-access-kpn4d\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.305459 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.305429 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qzn\" (UniqueName: \"kubernetes.io/projected/f9db17c4-a168-4dfb-a89a-88933fd6413b-kube-api-access-b4qzn\") pod \"network-check-source-6f78d59f99-2ngtk\" (UID: \"f9db17c4-a168-4dfb-a89a-88933fd6413b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:42.318481 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.318307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:42.337357 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.336378 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:42.419395 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.418575 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:42.552698 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.552620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:42.552909 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.552770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:42.552909 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.552784 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:42.552909 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.552807 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:42.553079 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.552912 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.552892805 +0000 UTC m=+32.134454264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:42.553079 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.552917 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 08:05:42.553079 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.552973 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls podName:163e10f5-9315-4873-82b8-f830ff3135ac nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.5529585 +0000 UTC m=+32.134519965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2n4f" (UID: "163e10f5-9315-4873-82b8-f830ff3135ac") : secret "samples-operator-tls" not found Nov 02 08:05:42.557348 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.557323 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qmp75"] Nov 02 08:05:42.557445 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.557362 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0e92c0_8b2e_439a_b72c_db5193d1c6bd.slice/crio-7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64 WatchSource:0}: Error finding container 7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64: Status 404 returned error can't find the container with id 7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64 Nov 02 08:05:42.560348 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.560295 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d14a7ee_7798_4273_9e07_68f3ad95b363.slice/crio-8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4 WatchSource:0}: Error finding container 8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4: Status 404 returned error can't find the container with id 8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4 Nov 02 08:05:42.561749 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.561727 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cbw8w"] Nov 02 08:05:42.566422 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.563037 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e166bd_4138_4d3d_9e2e_cbcca93e18f6.slice/crio-0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141 WatchSource:0}: Error finding container 0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141: Status 404 returned error can't find the container with id 0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141 Nov 02 08:05:42.566422 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.564502 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fr2k5"] Nov 02 08:05:42.630602 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.630503 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx"] Nov 02 08:05:42.633508 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.633465 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05387920_8354_45bf_85e3_513462f484bf.slice/crio-b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76 WatchSource:0}: Error finding container b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76: Status 404 returned error can't find the container with id b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76 Nov 02 08:05:42.653979 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.653948 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.654111 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.654016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:42.654212 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654129 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:42.654212 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654159 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:42.654212 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654162 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.654126944 +0000 UTC m=+32.235688389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : configmap references non-existent config key: service-ca.crt Nov 02 08:05:42.654212 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654199 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.654188004 +0000 UTC m=+32.235749462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.654215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.654250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.654285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654325 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654364 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls podName:611a6955-050e-4d16-b301-aa943a2f980e nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.654354353 +0000 UTC m=+32.235915798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-srmdt" (UID: "611a6955-050e-4d16-b301-aa943a2f980e") : secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654410 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654414 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:42.654427 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654434 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert podName:d6491032-ad9f-4d34-98b8-6fb3f03e4cd9 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.654426859 +0000 UTC m=+32.235988345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wtksb" (UID: "d6491032-ad9f-4d34-98b8-6fb3f03e4cd9") : secret "networking-console-plugin-cert" not found Nov 02 08:05:42.654799 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.654445 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.654439473 +0000 UTC m=+32.236000918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:42.680656 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.680625 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a3cb18_3e9a_4452_88bb_a8c0515db6fd.slice/crio-1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b WatchSource:0}: Error finding container 1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b: Status 404 returned error can't find the container with id 1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b Nov 02 08:05:42.681264 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.680715 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd"] Nov 02 08:05:42.715434 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.715407 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tspm9"] Nov 02 08:05:42.715593 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.715466 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438dc60b_6968_41c4_b974_cff36e50ab18.slice/crio-2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d WatchSource:0}: Error finding container 2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d: Status 404 returned error can't find the container with id 2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d Nov 02 08:05:42.717736 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.717711 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9db17c4_a168_4dfb_a89a_88933fd6413b.slice/crio-8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388 WatchSource:0}: Error finding container 8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388: Status 404 returned error can't find the container with id 8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388 Nov 02 08:05:42.718182 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.718161 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk"] Nov 02 08:05:42.728349 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.728326 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e8f7ea_8d81_4b80_a527_f6e228885382.slice/crio-f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e WatchSource:0}: Error finding container f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e: Status 404 returned error can't find the container with id f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e Nov 02 08:05:42.729358 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.729337 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-svhfk"] Nov 02 08:05:42.755650 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.755626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:42.755783 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.755766 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 08:05:42.755836 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:42.755825 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls podName:de009ad0-3b12-421f-a8bf-edfc92fd6b63 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:43.755806212 +0000 UTC m=+32.337367657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls") pod "dns-default-q49jt" (UID: "de009ad0-3b12-421f-a8bf-edfc92fd6b63") : secret "dns-default-metrics-tls" not found Nov 02 08:05:42.803035 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.803011 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38a56d4_3eca_4c4d_b7f9_fbd80403e692.slice/crio-5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac WatchSource:0}: Error finding container 5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac: Status 404 returned error can't find the container with id 5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac Nov 02 08:05:42.803752 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.803725 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-86sd2"] Nov 02 08:05:42.837637 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:42.837603 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8"] Nov 02 08:05:42.839011 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:42.838982 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86f89f2_60bc_44f1_b885_e33cc92bcd9f.slice/crio-59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070 WatchSource:0}: Error finding container 59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070: Status 404 returned error can't find the container with id 59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070 Nov 02 08:05:43.069316 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.069284 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" event={"ID":"f86f89f2-60bc-44f1-b885-e33cc92bcd9f","Type":"ContainerStarted","Data":"59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070"} Nov 02 08:05:43.070788 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.070759 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" event={"ID":"1d14a7ee-7798-4273-9e07-68f3ad95b363","Type":"ContainerStarted","Data":"ec240df4c14ab109ddd715b84fe23a8cdd397ea2e23f1d84e3418a1f09d0b20c"} Nov 02 08:05:43.070939 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.070806 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" event={"ID":"1d14a7ee-7798-4273-9e07-68f3ad95b363","Type":"ContainerStarted","Data":"8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4"} Nov 02 08:05:43.071874 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.071854 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" event={"ID":"f9db17c4-a168-4dfb-a89a-88933fd6413b","Type":"ContainerStarted","Data":"8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388"} Nov 02 08:05:43.073265 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.073240 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qmp75" event={"ID":"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd","Type":"ContainerStarted","Data":"dec30999e3bacffdd458275e94a2f3ed68eb86a13de3d4d7a7a4b869b2f0ad54"} Nov 02 08:05:43.073354 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.073284 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qmp75" event={"ID":"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd","Type":"ContainerStarted","Data":"7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64"} Nov 02 08:05:43.074334 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.074310 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-svhfk" event={"ID":"62e8f7ea-8d81-4b80-a527-f6e228885382","Type":"ContainerStarted","Data":"f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e"} Nov 02 08:05:43.075587 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.075564 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-86sd2" event={"ID":"e38a56d4-3eca-4c4d-b7f9-fbd80403e692","Type":"ContainerStarted","Data":"b078ba1c693cf03fc6ebe330a20bfd619205f18f894fdb6d392b19c68c2c573f"} Nov 02 08:05:43.075660 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.075604 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-86sd2" event={"ID":"e38a56d4-3eca-4c4d-b7f9-fbd80403e692","Type":"ContainerStarted","Data":"5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac"} Nov 02 08:05:43.076649 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.076623 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" event={"ID":"438dc60b-6968-41c4-b974-cff36e50ab18","Type":"ContainerStarted","Data":"2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d"} Nov 02 08:05:43.077742 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.077726 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" event={"ID":"19e166bd-4138-4d3d-9e2e-cbcca93e18f6","Type":"ContainerStarted","Data":"0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141"} Nov 02 08:05:43.078861 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.078830 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" event={"ID":"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd","Type":"ContainerStarted","Data":"1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b"} Nov 02 08:05:43.080002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.079978 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" event={"ID":"05387920-8354-45bf-85e3-513462f484bf","Type":"ContainerStarted","Data":"b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76"} Nov 02 08:05:43.382738 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.382641 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qmp75" podStartSLOduration=434.382625702 podStartE2EDuration="7m14.382625702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:43.381429485 +0000 UTC m=+31.962990952" watchObservedRunningTime="2026-11-02 08:05:43.382625702 +0000 UTC m=+31.964187165" Nov 02 08:05:43.562280 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.562076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:43.562473 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.562238 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 08:05:43.562473 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.562356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:43.562473 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.562407 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls podName:163e10f5-9315-4873-82b8-f830ff3135ac nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.562383783 +0000 UTC m=+34.143945232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2n4f" (UID: "163e10f5-9315-4873-82b8-f830ff3135ac") : secret "samples-operator-tls" not found Nov 02 08:05:43.562473 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.562451 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:43.562473 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.562461 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:43.562651 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.562502 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.562491605 +0000 UTC m=+34.144053050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:43.665304 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.665227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:43.665304 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.665265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.665307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.665337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.665373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665430 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.665408135 +0000 UTC m=+34.246969602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : configmap references non-existent config key: service-ca.crt Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665466 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665484 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:43.665497 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665492 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:43.665778 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665502 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert podName:d6491032-ad9f-4d34-98b8-6fb3f03e4cd9 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.665491636 +0000 UTC m=+34.247053081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wtksb" (UID: "d6491032-ad9f-4d34-98b8-6fb3f03e4cd9") : secret "networking-console-plugin-cert" not found Nov 02 08:05:43.665778 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665429 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:43.665778 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665585 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:43.665778 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665621 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls podName:611a6955-050e-4d16-b301-aa943a2f980e nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.665600812 +0000 UTC m=+34.247162264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-srmdt" (UID: "611a6955-050e-4d16-b301-aa943a2f980e") : secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:43.665778 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665640 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.665629838 +0000 UTC m=+34.247191282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:43.665778 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.665663 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.665654398 +0000 UTC m=+34.247215849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:43.721974 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.721884 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-86sd2" podStartSLOduration=434.721846654 podStartE2EDuration="7m14.721846654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:43.72096359 +0000 UTC m=+32.302525057" watchObservedRunningTime="2026-11-02 08:05:43.721846654 +0000 UTC m=+32.303408142" Nov 02 08:05:43.767088 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.767011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:43.767425 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.767404 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 08:05:43.767528 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:43.767487 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls podName:de009ad0-3b12-421f-a8bf-edfc92fd6b63 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.767468089 +0000 UTC m=+34.349029533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls") pod "dns-default-q49jt" (UID: "de009ad0-3b12-421f-a8bf-edfc92fd6b63") : secret "dns-default-metrics-tls" not found Nov 02 08:05:43.834807 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:43.833707 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" podStartSLOduration=434.833688874 podStartE2EDuration="7m14.833688874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:43.832983351 +0000 UTC m=+32.414544819" watchObservedRunningTime="2026-11-02 08:05:43.833688874 +0000 UTC m=+32.415250341" Nov 02 08:05:44.878297 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:44.877151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:44.878297 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:44.877215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:44.878297 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:44.877363 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 08:05:44.878297 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:44.877433 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:16.87741367 +0000 UTC m=+65.458975139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : secret "metrics-daemon-secret" not found Nov 02 08:05:44.900954 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:44.900896 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:44.938662 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:44.938405 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:44.977982 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:44.977939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:44.983209 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:44.983151 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:45.225686 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.225387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:45.327541 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.327462 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2phd8"] Nov 02 08:05:45.585082 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.585040 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:45.585308 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.585184 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:45.585308 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.585299 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:45.585308 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.585310 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:45.585479 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.585324 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 08:05:45.585479 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.585362 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.585349449 +0000 UTC m=+38.166910908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:45.585479 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.585376 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls podName:163e10f5-9315-4873-82b8-f830ff3135ac nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.585370333 +0000 UTC m=+38.166931777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2n4f" (UID: "163e10f5-9315-4873-82b8-f830ff3135ac") : secret "samples-operator-tls" not found Nov 02 08:05:45.659010 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.658972 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8d2bg"] Nov 02 08:05:45.665259 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:45.663370 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68e237e_b990_4625_ba3a_a02cdb285bd8.slice/crio-743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a WatchSource:0}: Error finding container 743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a: Status 404 returned error can't find the container with id 743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a Nov 02 08:05:45.686298 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.686260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:45.686447 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.686329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:45.686447 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.686390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:45.686572 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.686494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:45.686572 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.686522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:45.686808 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686652 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:45.686808 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686667 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:45.686808 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686728 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.686712034 +0000 UTC m=+38.268273502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:45.686808 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686805 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:45.687106 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686842 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls podName:611a6955-050e-4d16-b301-aa943a2f980e nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.686829977 +0000 UTC m=+38.268391438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-srmdt" (UID: "611a6955-050e-4d16-b301-aa943a2f980e") : secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:45.687106 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686895 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:45.687106 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686923 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.686913924 +0000 UTC m=+38.268475587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:45.687106 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.686971 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 08:05:45.687106 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.687002 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert podName:d6491032-ad9f-4d34-98b8-6fb3f03e4cd9 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.686993163 +0000 UTC m=+38.268554622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wtksb" (UID: "d6491032-ad9f-4d34-98b8-6fb3f03e4cd9") : secret "networking-console-plugin-cert" not found Nov 02 08:05:45.687106 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.687045 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.687038949 +0000 UTC m=+38.268600407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : configmap references non-existent config key: service-ca.crt Nov 02 08:05:45.787774 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:45.787676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:45.787949 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.787920 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 08:05:45.788020 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:45.787985 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls podName:de009ad0-3b12-421f-a8bf-edfc92fd6b63 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.787966634 +0000 UTC m=+38.369528093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls") pod "dns-default-q49jt" (UID: "de009ad0-3b12-421f-a8bf-edfc92fd6b63") : secret "dns-default-metrics-tls" not found Nov 02 08:05:46.103540 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:46.103442 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2phd8" event={"ID":"73d52e94-07b5-4a34-b8ec-74e17ad7d096","Type":"ContainerStarted","Data":"01ff3fed532fa7f60e66ad823804744d3abd1188531dcaae269de869444c916b"} Nov 02 08:05:46.106568 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:46.106526 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8d2bg" event={"ID":"f68e237e-b990-4625-ba3a-a02cdb285bd8","Type":"ContainerStarted","Data":"743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a"} Nov 02 08:05:46.110643 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:46.110592 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" event={"ID":"f86f89f2-60bc-44f1-b885-e33cc92bcd9f","Type":"ContainerStarted","Data":"4b2dc8e68b71c8935266bc27df73dfc2ea7b82d02b91f503cbb7a057eeac4580"} Nov 02 08:05:46.225337 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:46.225237 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" podStartSLOduration=406.708068296 podStartE2EDuration="6m49.225222859s" totalImagesPullingTime="2.517154563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:46.223952385 +0000 UTC m=+34.805513855" watchObservedRunningTime="2026-11-02 08:05:46.225222859 +0000 UTC m=+34.806784326" Nov 02 08:05:48.120412 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.120376 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" event={"ID":"05387920-8354-45bf-85e3-513462f484bf","Type":"ContainerStarted","Data":"194264e54169b934c2f347dcd469b8426cf7e450fc5a03d08b2f651dc4e09ee2"} Nov 02 08:05:48.124308 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.124272 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/0.log" Nov 02 08:05:48.124451 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.124310 2582 generic.go:356] "Generic (PLEG): container finished" podID="438dc60b-6968-41c4-b974-cff36e50ab18" containerID="6ba113d68ce0dd52c359eff71df97262e8bf48691bf7c17a75dd63997bdd481a" exitCode=255 Nov 02 08:05:48.124451 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.124385 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" event={"ID":"438dc60b-6968-41c4-b974-cff36e50ab18","Type":"ContainerDied","Data":"6ba113d68ce0dd52c359eff71df97262e8bf48691bf7c17a75dd63997bdd481a"} Nov 02 08:05:48.125759 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.124815 2582 scope.go:122] "RemoveContainer" containerID="6ba113d68ce0dd52c359eff71df97262e8bf48691bf7c17a75dd63997bdd481a" Nov 02 08:05:48.127179 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.127120 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" event={"ID":"19e166bd-4138-4d3d-9e2e-cbcca93e18f6","Type":"ContainerStarted","Data":"9f6083b4f7eace095c0dbe458476bee0b1b9b4e10ca9d5f04eaa9d7a516a10bd"} Nov 02 08:05:48.128996 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.128831 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" event={"ID":"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd","Type":"ContainerStarted","Data":"adc2e405949023b91e3e03cea945305eca41257b9703660bd3554bde0a761b82"} Nov 02 08:05:48.235059 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.234205 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" podStartSLOduration=410.18060646 podStartE2EDuration="6m55.234185165s" totalImagesPullingTime="5.053578705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:48.233495216 +0000 UTC m=+36.815056683" watchObservedRunningTime="2026-11-02 08:05:48.234185165 +0000 UTC m=+36.815746635" Nov 02 08:05:48.353676 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.353615 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" podStartSLOduration=410.534920383 podStartE2EDuration="6m55.353595297s" totalImagesPullingTime="4.818674914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:48.35285532 +0000 UTC m=+36.934416786" watchObservedRunningTime="2026-11-02 08:05:48.353595297 +0000 UTC m=+36.935156765" Nov 02 08:05:48.592991 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.588865 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" podStartSLOduration=407.657937389 podStartE2EDuration="6m52.588843932s" totalImagesPullingTime="4.930906543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:48.587613897 +0000 UTC m=+37.169175365" watchObservedRunningTime="2026-11-02 08:05:48.588843932 +0000 UTC m=+37.170405404" Nov 02 08:05:48.831748 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.831725 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48rm2"] Nov 02 08:05:48.836021 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.835399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:48.893930 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.893902 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 08:05:48.894692 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.894670 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 08:05:48.894828 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.894811 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9tsd7\"" Nov 02 08:05:48.899990 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:48.899963 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48rm2"] Nov 02 08:05:49.024481 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.024447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.024648 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.024533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rlh\" (UniqueName: \"kubernetes.io/projected/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-api-access-22rlh\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.024648 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.024561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.024648 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.024613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5d3993c-5c24-4912-9113-f64e2ca954b3-crio-socket\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.024806 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.024663 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d3993c-5c24-4912-9113-f64e2ca954b3-data-volume\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.026295 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.026269 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t29pm-98s5b"] Nov 02 08:05:49.029008 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.028983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.084528 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.084491 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t29pm-98s5b"] Nov 02 08:05:49.125867 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.125782 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d3993c-5c24-4912-9113-f64e2ca954b3-data-volume\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.125867 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.125865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.125942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rlh\" (UniqueName: \"kubernetes.io/projected/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-api-access-22rlh\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.126249 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126255 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d3993c-5c24-4912-9113-f64e2ca954b3-data-volume\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.126308 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.126324 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.626304077 +0000 UTC m=+38.207865541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:49.126477 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.126452 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:49.626375226 +0000 UTC m=+38.207936686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : configmap "kube-rbac-proxy" not found Nov 02 08:05:49.126983 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5d3993c-5c24-4912-9113-f64e2ca954b3-crio-socket\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.126983 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126546 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.126983 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.126722 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5d3993c-5c24-4912-9113-f64e2ca954b3-crio-socket\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.135176 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.135124 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-svhfk" event={"ID":"62e8f7ea-8d81-4b80-a527-f6e228885382","Type":"ContainerStarted","Data":"3eaaa5fd77374862c7539a458f2816f9ee11462177d08b3ea5a41f3b7528469f"} Nov 02 08:05:49.137383 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.137362 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:05:49.137873 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.137857 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/0.log" Nov 02 08:05:49.137964 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.137892 2582 generic.go:356] "Generic (PLEG): container finished" podID="438dc60b-6968-41c4-b974-cff36e50ab18" containerID="96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8" exitCode=255 Nov 02 08:05:49.138130 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.138106 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" event={"ID":"438dc60b-6968-41c4-b974-cff36e50ab18","Type":"ContainerDied","Data":"96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8"} Nov 02 08:05:49.138268 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.138201 2582 scope.go:122] "RemoveContainer" containerID="6ba113d68ce0dd52c359eff71df97262e8bf48691bf7c17a75dd63997bdd481a" Nov 02 08:05:49.138646 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.138627 2582 scope.go:122] "RemoveContainer" containerID="96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8" Nov 02 08:05:49.138851 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.138819 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tspm9_openshift-console-operator(438dc60b-6968-41c4-b974-cff36e50ab18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" podUID="438dc60b-6968-41c4-b974-cff36e50ab18" Nov 02 08:05:49.147693 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.147597 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jw25f\"" Nov 02 08:05:49.186210 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.186178 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rlh\" (UniqueName: \"kubernetes.io/projected/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-api-access-22rlh\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.228457 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.228037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.228457 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.228113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.228457 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.228261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.228457 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.228303 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.228874 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.228822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.229371 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.229346 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.229485 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.229457 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.262913 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.262853 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-svhfk" podStartSLOduration=2.189848086 podStartE2EDuration="8.26283504s" totalImagesPullingTime="6.072986954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:49.260918147 +0000 UTC m=+37.842479617" watchObservedRunningTime="2026-11-02 08:05:49.26283504 +0000 UTC m=+37.844396508" Nov 02 08:05:49.269588 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.269561 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm"] Nov 02 08:05:49.273053 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.272693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:49.278267 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.278244 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs"] Nov 02 08:05:49.280279 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.280259 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:49.293851 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.293828 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.326676 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.326094 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm"] Nov 02 08:05:49.326676 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.326401 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:49.326676 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.326534 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5nd8v\"" Nov 02 08:05:49.326676 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.326637 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 08:05:49.341617 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.341581 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs"] Nov 02 08:05:49.344896 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.344782 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:49.432002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.431332 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lrk\" (UniqueName: \"kubernetes.io/projected/f25ae8a9-a299-4fdd-a752-6589f9f9b93a-kube-api-access-64lrk\") pod \"migrator-66df5d496b-k9qrm\" (UID: \"f25ae8a9-a299-4fdd-a752-6589f9f9b93a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:49.432002 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.431417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvx5\" (UniqueName: \"kubernetes.io/projected/7e919b41-8ea7-4e4f-a087-b5889c997875-kube-api-access-vtvx5\") pod \"migrator-66df5d496b-tczzs\" (UID: \"7e919b41-8ea7-4e4f-a087-b5889c997875\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:49.532630 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.532584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lrk\" (UniqueName: \"kubernetes.io/projected/f25ae8a9-a299-4fdd-a752-6589f9f9b93a-kube-api-access-64lrk\") pod \"migrator-66df5d496b-k9qrm\" (UID: \"f25ae8a9-a299-4fdd-a752-6589f9f9b93a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:49.532821 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.532682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvx5\" (UniqueName: \"kubernetes.io/projected/7e919b41-8ea7-4e4f-a087-b5889c997875-kube-api-access-vtvx5\") pod \"migrator-66df5d496b-tczzs\" (UID: \"7e919b41-8ea7-4e4f-a087-b5889c997875\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:49.597662 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.597632 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvx5\" (UniqueName: \"kubernetes.io/projected/7e919b41-8ea7-4e4f-a087-b5889c997875-kube-api-access-vtvx5\") pod \"migrator-66df5d496b-tczzs\" (UID: \"7e919b41-8ea7-4e4f-a087-b5889c997875\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:49.598127 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.598095 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lrk\" (UniqueName: \"kubernetes.io/projected/f25ae8a9-a299-4fdd-a752-6589f9f9b93a-kube-api-access-64lrk\") pod \"migrator-66df5d496b-k9qrm\" (UID: \"f25ae8a9-a299-4fdd-a752-6589f9f9b93a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:49.615999 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.615973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:49.633645 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.633590 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.633731 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.633703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:49.633779 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.633742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:49.633861 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.633844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:49.633973 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.633869 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:49.633973 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.633934 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:50.633914376 +0000 UTC m=+39.215475830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:49.633973 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.633942 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 08:05:49.634092 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.633979 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls podName:163e10f5-9315-4873-82b8-f830ff3135ac nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.633969275 +0000 UTC m=+46.215530720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2n4f" (UID: "163e10f5-9315-4873-82b8-f830ff3135ac") : secret "samples-operator-tls" not found Nov 02 08:05:49.634092 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.633991 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 08:05:49.634092 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.634022 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:49.634092 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.634028 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:50.634016124 +0000 UTC m=+39.215577573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : configmap "kube-rbac-proxy" not found Nov 02 08:05:49.634092 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.634030 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:49.634092 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.634083 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.634064309 +0000 UTC m=+46.215625780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:49.723866 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:49.723763 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86dbbbd_8799_4701_9fa5_831d1388c9d7.slice/crio-fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd WatchSource:0}: Error finding container fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd: Status 404 returned error can't find the container with id fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd Nov 02 08:05:49.726342 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.726200 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t29pm-98s5b"] Nov 02 08:05:49.735026 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.735001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:49.735167 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.735035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:49.735167 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.735067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:49.735167 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.735098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:49.735167 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.735125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:49.735361 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735252 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 08:05:49.735361 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735299 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert podName:d6491032-ad9f-4d34-98b8-6fb3f03e4cd9 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.735286856 +0000 UTC m=+46.316848301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wtksb" (UID: "d6491032-ad9f-4d34-98b8-6fb3f03e4cd9") : secret "networking-console-plugin-cert" not found Nov 02 08:05:49.735361 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735347 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.735340618 +0000 UTC m=+46.316902063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : configmap references non-existent config key: service-ca.crt Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735387 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735394 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735415 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.735409132 +0000 UTC m=+46.316970577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735447 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735466 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls podName:611a6955-050e-4d16-b301-aa943a2f980e nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.73546116 +0000 UTC m=+46.317022606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-srmdt" (UID: "611a6955-050e-4d16-b301-aa943a2f980e") : secret "cluster-monitoring-operator-tls" not found Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735495 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:49.735510 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.735512 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.735506747 +0000 UTC m=+46.317068192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:49.836361 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.835917 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:49.836361 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.836258 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 08:05:49.836361 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:49.836324 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls podName:de009ad0-3b12-421f-a8bf-edfc92fd6b63 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.836306747 +0000 UTC m=+46.417868205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls") pod "dns-default-q49jt" (UID: "de009ad0-3b12-421f-a8bf-edfc92fd6b63") : secret "dns-default-metrics-tls" not found Nov 02 08:05:49.893502 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.893447 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:49.988996 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:49.988966 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs"] Nov 02 08:05:49.989583 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:49.989554 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e919b41_8ea7_4e4f_a087_b5889c997875.slice/crio-28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c WatchSource:0}: Error finding container 28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c: Status 404 returned error can't find the container with id 28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c Nov 02 08:05:50.143412 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.143367 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" event={"ID":"7e919b41-8ea7-4e4f-a087-b5889c997875","Type":"ContainerStarted","Data":"28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c"} Nov 02 08:05:50.145606 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.145581 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:05:50.146013 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.145990 2582 scope.go:122] "RemoveContainer" containerID="96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8" Nov 02 08:05:50.146539 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:50.146510 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tspm9_openshift-console-operator(438dc60b-6968-41c4-b974-cff36e50ab18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" podUID="438dc60b-6968-41c4-b974-cff36e50ab18" Nov 02 08:05:50.148072 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.148049 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8d2bg" event={"ID":"f68e237e-b990-4625-ba3a-a02cdb285bd8","Type":"ContainerStarted","Data":"35d47f3c68ce4392f80bd7d984d66401ee773916fe91a9e35717b7a5e323bbea"} Nov 02 08:05:50.148242 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.148128 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:50.151261 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.151230 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t29pm-98s5b" event={"ID":"d86dbbbd-8799-4701-9fa5-831d1388c9d7","Type":"ContainerStarted","Data":"41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e"} Nov 02 08:05:50.151357 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.151297 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t29pm-98s5b" event={"ID":"d86dbbbd-8799-4701-9fa5-831d1388c9d7","Type":"ContainerStarted","Data":"fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd"} Nov 02 08:05:50.152844 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.152819 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" event={"ID":"f9db17c4-a168-4dfb-a89a-88933fd6413b","Type":"ContainerStarted","Data":"11005f5c4121c49e06ee3b60a88c6d56ac14845ff96be112e021fe83947a1ecd"} Nov 02 08:05:50.262177 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:50.258765 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25ae8a9_a299_4fdd_a752_6589f9f9b93a.slice/crio-a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1 WatchSource:0}: Error finding container a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1: Status 404 returned error can't find the container with id a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1 Nov 02 08:05:50.262177 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.259787 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm"] Nov 02 08:05:50.374195 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.374114 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" podStartSLOduration=404.520484584 podStartE2EDuration="6m51.374100198s" totalImagesPullingTime="6.853615614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:50.373279369 +0000 UTC m=+38.954840850" watchObservedRunningTime="2026-11-02 08:05:50.374100198 +0000 UTC m=+38.955661664" Nov 02 08:05:50.487323 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.487275 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t29pm-98s5b" podStartSLOduration=2.487260103 podStartE2EDuration="2.487260103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:50.486199353 +0000 UTC m=+39.067760820" watchObservedRunningTime="2026-11-02 08:05:50.487260103 +0000 UTC m=+39.068821570" Nov 02 08:05:50.583224 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.583134 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lw557"] Nov 02 08:05:50.585789 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.585766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.602923 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.602883 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8d2bg" podStartSLOduration=34.587569853 podStartE2EDuration="38.60286961s" totalImagesPullingTime="4.015299757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:50.602220887 +0000 UTC m=+39.183782366" watchObservedRunningTime="2026-11-02 08:05:50.60286961 +0000 UTC m=+39.184431077" Nov 02 08:05:50.641632 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.641602 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kvw72\"" Nov 02 08:05:50.641805 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.641681 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 08:05:50.644748 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.644695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6k26\" (UniqueName: \"kubernetes.io/projected/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-kube-api-access-c6k26\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.644878 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.644834 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lw557"] Nov 02 08:05:50.645472 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.645437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:50.645579 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.645501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-key\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.645629 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.645583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:50.645629 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:50.645609 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:50.645716 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.645627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-cabundle\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.645716 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:50.645689 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:52.645671483 +0000 UTC m=+41.227232954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:50.645805 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:50.645760 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 08:05:50.645805 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:50.645795 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:52.645783616 +0000 UTC m=+41.227345078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : configmap "kube-rbac-proxy" not found Nov 02 08:05:50.690105 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.690060 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 08:05:50.690105 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.690102 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 08:05:50.690312 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.690119 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 08:05:50.746377 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.746337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k26\" (UniqueName: \"kubernetes.io/projected/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-kube-api-access-c6k26\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.746586 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.746481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-key\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.746647 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.746602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-cabundle\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.747525 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.747496 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-cabundle\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.749455 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.749417 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-key\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.828994 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.828962 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k26\" (UniqueName: \"kubernetes.io/projected/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-kube-api-access-c6k26\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:50.900169 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:50.900064 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:51.166419 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:51.165730 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" event={"ID":"f25ae8a9-a299-4fdd-a752-6589f9f9b93a","Type":"ContainerStarted","Data":"a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1"} Nov 02 08:05:51.282410 ip-10-0-139-247 kubenswrapper[2582]: W1102 08:05:51.281235 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff50b9eb_13e4_4e99_8dcd_b14db45dff7d.slice/crio-0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5 WatchSource:0}: Error finding container 0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5: Status 404 returned error can't find the container with id 0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5 Nov 02 08:05:51.282410 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:51.281685 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lw557"] Nov 02 08:05:52.171904 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.171822 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" event={"ID":"7e919b41-8ea7-4e4f-a087-b5889c997875","Type":"ContainerStarted","Data":"dfc64568cb5c3893a76f3cfe8c0209fee1fd751fd6be48913a8c3a6dc737caff"} Nov 02 08:05:52.173518 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.173485 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" event={"ID":"f25ae8a9-a299-4fdd-a752-6589f9f9b93a","Type":"ContainerStarted","Data":"b195ff17fc66dce132790ea46ac31e539cefcefc093b85955f988261931670e2"} Nov 02 08:05:52.175092 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.175066 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" event={"ID":"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d","Type":"ContainerStarted","Data":"31f4314c53b58c50d1c5f95f7417f0d5b41dd00636d448a02d70084022f0292a"} Nov 02 08:05:52.175205 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.175106 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" event={"ID":"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d","Type":"ContainerStarted","Data":"0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5"} Nov 02 08:05:52.287733 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.287672 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" podStartSLOduration=1.371838144 podStartE2EDuration="3.287658673s" totalImagesPullingTime="1.915820529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:52.287057352 +0000 UTC m=+40.868618821" watchObservedRunningTime="2026-11-02 08:05:52.287658673 +0000 UTC m=+40.869220156" Nov 02 08:05:52.301104 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.301068 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:52.301314 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.301155 2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:52.301610 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.301577 2582 scope.go:122] "RemoveContainer" containerID="96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8" Nov 02 08:05:52.301830 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:52.301798 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tspm9_openshift-console-operator(438dc60b-6968-41c4-b974-cff36e50ab18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" podUID="438dc60b-6968-41c4-b974-cff36e50ab18" Nov 02 08:05:52.403365 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.403298 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" podStartSLOduration=1.629461707 podStartE2EDuration="3.403279612s" totalImagesPullingTime="1.773817905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:52.401380746 +0000 UTC m=+40.982942214" watchObservedRunningTime="2026-11-02 08:05:52.403279612 +0000 UTC m=+40.984841079" Nov 02 08:05:52.516430 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.516377 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" podStartSLOduration=2.516360466 podStartE2EDuration="2.516360466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:52.515313734 +0000 UTC m=+41.096875201" watchObservedRunningTime="2026-11-02 08:05:52.516360466 +0000 UTC m=+41.097921932" Nov 02 08:05:52.669332 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.669298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:52.669439 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.669407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:52.669476 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:52.669447 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:52.669516 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:52.669509 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:56.669492839 +0000 UTC m=+45.251054304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:52.669555 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:52.669519 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 08:05:52.669612 ip-10-0-139-247 kubenswrapper[2582]: E1102 08:05:52.669565 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:56.669551039 +0000 UTC m=+45.251112484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : configmap "kube-rbac-proxy" not found Nov 02 08:05:52.839022 ip-10-0-139-247 kubenswrapper[2582]: I1102 08:05:52.838787 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:52.838958 ip-10-0-139-247 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 08:05:52.851418 ip-10-0-139-247 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 08:05:52.851592 ip-10-0-139-247 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 08:05:52.851781 ip-10-0-139-247 systemd[1]: kubelet.service: Consumed 3.302s CPU time. Nov 02 08:05:52.865801 ip-10-0-139-247 systemd[1]: Starting Kubernetes Kubelet... Nov 02 08:05:52.940865 ip-10-0-139-247 kubenswrapper[5055]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:52.940865 ip-10-0-139-247 kubenswrapper[5055]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 08:05:52.940865 ip-10-0-139-247 kubenswrapper[5055]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:52.940865 ip-10-0-139-247 kubenswrapper[5055]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:52.944201 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944084 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.944201 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944200 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944207 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944212 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944217 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944222 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944226 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944230 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944234 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944239 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944243 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944247 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944251 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944256 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944261 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944265 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944269 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944273 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944278 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944282 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.944334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944286 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944290 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944295 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944299 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944303 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944307 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944311 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944316 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944321 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944325 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944332 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944336 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944341 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944345 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944349 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944354 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944358 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944362 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944366 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944371 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.945097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944375 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944379 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944383 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944387 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944391 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944395 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944400 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944404 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944408 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944412 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944416 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944420 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944428 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944434 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944440 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944445 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944449 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944454 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944458 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.945906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944462 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944466 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944470 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944476 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944480 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944487 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944492 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944498 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944503 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944507 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944511 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944515 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944519 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944524 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944528 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944534 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944538 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944542 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944546 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.946726 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944550 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944554 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944558 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944562 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944566 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944571 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944575 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944579 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944584 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944588 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944593 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944597 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944601 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944605 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944609 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944613 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944617 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944621 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944625 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944630 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.947854 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944634 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944638 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944642 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944646 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944650 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944654 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944658 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.944663 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945825 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945834 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945839 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945843 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945848 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945852 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945856 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945860 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945865 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945872 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945878 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.950361 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945883 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945888 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945893 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945897 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945902 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945906 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945910 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945914 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945919 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945923 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945928 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945932 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945936 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945940 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945944 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945950 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945955 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945959 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945963 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.951131 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945967 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945971 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945976 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945981 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945985 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945989 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945993 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.945997 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946001 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946006 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946010 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946013 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946018 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946022 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946026 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946031 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946037 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946042 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946047 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946051 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.963233 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946056 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946060 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946064 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946075 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946079 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946084 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946089 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946093 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946097 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946101 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946106 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946111 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946115 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946120 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946124 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946129 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946133 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946152 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946156 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946160 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.964009 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946165 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946169 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946173 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946177 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946182 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946186 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946192 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946197 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946201 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946205 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946209 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946214 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946218 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946222 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946226 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946230 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946235 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946239 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946243 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946247 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.964542 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946252 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946256 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946260 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946264 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946268 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946272 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946277 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946281 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946298 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946304 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946308 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946312 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946316 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946320 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946324 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.946328 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947099 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947107 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947113 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947117 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.965074 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947122 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947126 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947130 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947135 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947154 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947158 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947163 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947167 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947171 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947176 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947180 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947184 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947188 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947192 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947199 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947204 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947208 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947212 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947217 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.965837 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947221 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947226 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947230 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947234 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947238 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947242 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947246 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947250 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947254 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947261 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947267 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947273 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947278 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947282 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947286 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947290 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947294 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947301 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947305 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.966352 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947309 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947313 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947318 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947322 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947326 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947330 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947334 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947338 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947342 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947346 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947350 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947354 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947358 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947362 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947366 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947371 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947377 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947381 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947385 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947389 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.966807 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947393 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947397 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947401 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947405 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947411 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947415 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947419 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947423 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947428 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947432 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947438 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947442 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947446 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947451 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947455 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947459 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947463 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947467 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947471 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947476 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.967415 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947480 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947484 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947489 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947493 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947497 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947501 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947507 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947511 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947515 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947519 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947523 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947527 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947531 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947535 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947539 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947543 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947549 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947553 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947558 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947562 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.968239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947566 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947570 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947576 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.947581 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947725 5055 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947735 5055 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947743 5055 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947750 5055 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947757 5055 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947762 5055 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947768 5055 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947774 5055 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947779 5055 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947784 5055 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947790 5055 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947795 5055 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947800 5055 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947804 5055 flags.go:64] FLAG: --cgroup-root="" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947809 5055 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947814 5055 flags.go:64] FLAG: --client-ca-file="" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947818 5055 flags.go:64] FLAG: --cloud-provider="external" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947824 5055 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947831 5055 flags.go:64] FLAG: --cluster-domain="" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947835 5055 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 08:05:52.969036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947840 5055 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947846 5055 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947851 5055 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947857 5055 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947864 5055 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947869 5055 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947874 5055 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947879 5055 flags.go:64] FLAG: --contention-profiling="false" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947883 5055 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947888 5055 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947894 5055 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947899 5055 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947905 5055 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947910 5055 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947915 5055 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947919 5055 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947924 5055 flags.go:64] FLAG: --enable-server="true" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947929 5055 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947935 5055 flags.go:64] FLAG: --event-burst="100" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947940 5055 flags.go:64] FLAG: --event-qps="50" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947946 5055 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947951 5055 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947955 5055 flags.go:64] FLAG: --eviction-hard="" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947962 5055 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 08:05:52.969988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947967 5055 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947971 5055 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947976 5055 flags.go:64] FLAG: --eviction-soft="" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947981 5055 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947986 5055 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947990 5055 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.947995 5055 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948000 5055 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948005 5055 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948009 5055 flags.go:64] FLAG: --feature-gates="" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948015 5055 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948020 5055 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948027 5055 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948032 5055 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948037 5055 flags.go:64] FLAG: --healthz-port="10248" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948042 5055 flags.go:64] FLAG: --help="false" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948047 5055 flags.go:64] FLAG: --hostname-override="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948052 5055 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948057 5055 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948062 5055 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948067 5055 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948073 5055 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948078 5055 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 08:05:52.971061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948082 5055 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948087 5055 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948091 5055 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948096 5055 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948101 5055 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948106 5055 flags.go:64] FLAG: --kube-reserved="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948111 5055 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948115 5055 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948120 5055 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948125 5055 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948129 5055 flags.go:64] FLAG: --lock-file="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948134 5055 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948151 5055 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948157 5055 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948164 5055 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948168 5055 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948174 5055 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948179 5055 flags.go:64] FLAG: --logging-format="text" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948183 5055 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948189 5055 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948193 5055 flags.go:64] FLAG: --manifest-url="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948200 5055 flags.go:64] FLAG: --manifest-url-header="" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948206 5055 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948211 5055 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948217 5055 flags.go:64] FLAG: --max-pods="110" Nov 02 08:05:52.972443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948222 5055 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948227 5055 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948232 5055 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948237 5055 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948242 5055 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948246 5055 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948251 5055 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948261 5055 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948266 5055 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948271 5055 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948276 5055 flags.go:64] FLAG: --pod-cidr="" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948280 5055 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948285 5055 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948290 5055 flags.go:64] FLAG: --pods-per-core="0" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948295 5055 flags.go:64] FLAG: --port="10250" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948299 5055 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948304 5055 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-032ef74f3e0e41b96" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948309 5055 flags.go:64] FLAG: --qos-reserved="" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948314 5055 flags.go:64] FLAG: --read-only-port="10255" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948318 5055 flags.go:64] FLAG: --register-node="true" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948323 5055 flags.go:64] FLAG: --register-with-taints="" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948329 5055 flags.go:64] FLAG: --registry-burst="10" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948333 5055 flags.go:64] FLAG: --registry-qps="5" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948338 5055 flags.go:64] FLAG: --reserved-cpus="" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948343 5055 flags.go:64] FLAG: --reserved-memory="" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948349 5055 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 08:05:52.973529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948354 5055 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948359 5055 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948365 5055 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948370 5055 flags.go:64] FLAG: --runonce="false" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948375 5055 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948380 5055 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948387 5055 flags.go:64] FLAG: --seccomp-default="false" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948392 5055 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948396 5055 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948401 5055 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948407 5055 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948411 5055 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948416 5055 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948421 5055 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948425 5055 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948430 5055 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948435 5055 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948440 5055 flags.go:64] FLAG: --system-cgroups="" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948445 5055 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948453 5055 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948458 5055 flags.go:64] FLAG: --tls-cert-file="" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948463 5055 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948469 5055 flags.go:64] FLAG: --tls-min-version="" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948473 5055 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948478 5055 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 08:05:52.974689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948482 5055 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948487 5055 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948491 5055 flags.go:64] FLAG: --v="2" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948498 5055 flags.go:64] FLAG: --version="false" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948504 5055 flags.go:64] FLAG: --vmodule="" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948510 5055 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.948515 5055 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948719 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948725 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948731 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948735 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948739 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948744 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948749 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948753 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948757 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948762 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948766 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948771 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948775 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948779 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.975713 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948784 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948788 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948792 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948796 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948801 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948805 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948809 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948813 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948818 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948822 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948826 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948830 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948838 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948843 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948847 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948851 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948855 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948859 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948864 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948868 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.976607 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948873 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948877 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948882 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948886 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948891 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948895 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948900 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948904 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948908 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948912 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948917 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948921 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948925 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948929 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948934 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948940 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948946 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948951 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948957 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948961 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.977491 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948965 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948970 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948974 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948978 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948982 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948987 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948991 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948995 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.948999 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949004 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949008 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949013 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949019 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949023 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949028 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949032 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949037 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949042 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949046 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949051 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.978267 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949055 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949059 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949064 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949068 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949072 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949077 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949080 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949087 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949093 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949097 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949101 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949105 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949109 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949114 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949118 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949122 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949127 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949131 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.979110 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949135 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949152 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949157 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949162 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949166 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949170 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949178 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949182 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949186 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949190 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949196 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949200 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949205 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949209 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.949249 5055 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949301 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.984722 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949308 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949314 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949318 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949323 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949327 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949331 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949335 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949340 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949344 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949349 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949353 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949358 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949362 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949366 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949370 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949374 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949378 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949383 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949387 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949391 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.985430 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949395 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949402 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949409 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949416 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949420 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949424 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949430 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949434 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949439 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949443 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949447 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949451 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949455 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949460 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949464 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949468 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949473 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949477 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.986258 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949481 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949485 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949489 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949494 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949498 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949502 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949507 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949511 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949515 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949519 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949523 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949527 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949532 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949536 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949540 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949544 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949548 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949554 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949559 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.987014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949563 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949568 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949573 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949577 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949581 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949585 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949589 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949594 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949598 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949602 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949606 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949611 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949615 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949619 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949623 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949627 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949631 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949636 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949640 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949644 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.987812 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949648 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949652 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949657 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949661 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949665 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949669 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949673 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949677 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949681 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949686 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949691 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949696 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949700 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949706 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949710 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949714 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949719 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949723 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949729 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949734 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.988638 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949739 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949743 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949748 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949752 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949756 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949760 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949766 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.949770 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.954638 5055 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.954655 5055 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.954674 5055 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.954680 5055 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954740 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954745 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954748 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954751 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954754 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954756 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954759 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954761 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.989469 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954764 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954767 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954770 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954773 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954776 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954778 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954781 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954784 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954789 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954792 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954796 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954800 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954802 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954805 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954808 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954811 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954814 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954819 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954822 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954825 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.990290 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954827 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954836 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954839 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954842 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954844 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954847 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954850 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954853 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954856 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954858 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954861 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954864 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954866 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954869 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954871 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954874 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954877 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954879 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954882 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.991090 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954885 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954887 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954890 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954893 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954897 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954900 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954902 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954905 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954908 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954910 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954913 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954916 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954919 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954921 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954924 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954932 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954935 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954938 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954940 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954943 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.991873 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954946 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954948 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954951 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954954 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954957 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954959 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954962 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954965 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954970 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954972 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954975 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954978 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954980 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954983 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954986 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954988 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954992 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954994 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.954997 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955000 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.993768 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955003 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955005 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955008 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955011 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955013 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955016 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955019 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955021 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955029 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955032 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955035 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955038 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955040 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955043 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955045 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955048 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955050 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955053 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.994596 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955055 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.955084 5055 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955300 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955306 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955309 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955312 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955315 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955318 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955321 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955323 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955326 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955329 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955332 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955334 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955337 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955340 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:52.995334 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955342 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955345 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955348 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955350 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955353 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955356 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955358 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955367 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955370 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955372 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955375 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955377 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955381 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955383 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955386 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955389 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955391 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955394 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955398 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955401 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:52.995986 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955403 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955406 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955409 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955411 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955414 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955417 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955419 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955422 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955425 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955428 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955431 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955433 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955436 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955439 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955441 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955444 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955447 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955449 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955452 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955454 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:52.996791 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955462 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955465 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955468 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955470 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955473 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955475 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955480 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955482 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955485 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955488 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955492 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955494 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955497 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955500 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955506 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955509 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955512 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955515 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955517 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955520 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:52.997758 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955523 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955526 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955529 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955531 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955534 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955537 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955539 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955542 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955545 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955547 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955550 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955553 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955556 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955565 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955568 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955571 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955573 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955576 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955580 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:52.998614 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955583 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955586 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955589 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955593 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955596 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955598 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955601 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955605 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955609 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955613 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955616 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955620 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955623 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.955648 5055 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955703 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:52.999435 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955708 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955711 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955714 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955717 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955719 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955722 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955725 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955728 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955731 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955733 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955736 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955738 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955747 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955750 5055 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955753 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955755 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955758 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955762 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955765 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955768 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:53.000059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955772 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955775 5055 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955778 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955781 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955784 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955786 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955789 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955792 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955795 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955797 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955800 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955803 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955805 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955808 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955811 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955814 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955816 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955819 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955821 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955824 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:53.000937 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955827 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955829 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955832 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955835 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955837 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955845 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955848 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955851 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955853 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955857 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955860 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955863 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955867 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955869 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955872 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955875 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955878 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955881 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955883 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955886 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:53.001906 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955889 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955891 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955894 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955897 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955900 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955903 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955905 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955908 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955911 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955913 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955916 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955919 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955921 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955925 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955928 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955930 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955933 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955936 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955944 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:53.002777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955947 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955949 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955955 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955958 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955962 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955968 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955971 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955974 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955977 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955980 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955982 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955985 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955988 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955991 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955994 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955996 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.955999 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956002 5055 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956005 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:53.004082 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956007 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956011 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956013 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956016 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956019 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956021 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:52.956024 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.956085 5055 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.960046 5055 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.960119 5055 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.961030 5055 server.go:1014] "Starting client certificate rotation" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.961174 5055 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.961212 5055 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 08:00:11 +0000 UTC" deadline="2028-08-03 22:27:43.899361262 +0000 UTC" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.961248 5055 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15374h21m50.938114838s" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.962000 5055 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.977131 5055 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:53.004924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.982520 5055 log.go:25] "Validated CRI v1 runtime API" Nov 02 08:05:53.005599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.985260 5055 log.go:25] "Validated CRI v1 image API" Nov 02 08:05:53.005599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.986313 5055 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 08:05:53.005599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.989284 5055 fs.go:135] Filesystem UUIDs: map[34235341-2beb-495d-94d8-8db3bd8fa51a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c345fb7-3f99-4983-97a0-0f6a908df7f3:/dev/nvme0n1p4] Nov 02 08:05:53.005770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.989311 5055 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01ff3fed532fa7f60e66ad823804744d3abd1188531dcaae269de869444c916b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01ff3fed532fa7f60e66ad823804744d3abd1188531dcaae269de869444c916b/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2058ac1dfa1098d746ac969bcaf0d27ef177bce954adf7195debaf39d8c7613c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2058ac1dfa1098d746ac969bcaf0d27ef177bce954adf7195debaf39d8c7613c/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05387920-8354-45bf-85e3-513462f484bf/volumes/kubernetes.io~projected/kube-api-access-dx6v6:{mountpoint:/var/lib/kubelet/pods/05387920-8354-45bf-85e3-513462f484bf/volumes/kubernetes.io~projected/kube-api-access-dx6v6 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05387920-8354-45bf-85e3-513462f484bf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/05387920-8354-45bf-85e3-513462f484bf/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volumes/kubernetes.io~projected/kube-api-access-tznrq:{mountpoint:/var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volumes/kubernetes.io~projected/kube-api-access-tznrq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/163e10f5-9315-4873-82b8-f830ff3135ac/volumes/kubernetes.io~projected/kube-api-access-9wkgx:{mountpoint:/var/lib/kubelet/pods/163e10f5-9315-4873-82b8-f830ff3135ac/volumes/kubernetes.io~projected/kube-api-access-9wkgx major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e166bd-4138-4d3d-9e2e-cbcca93e18f6/volumes/kubernetes.io~projected/kube-api-access-8z2w4:{mountpoint:/var/lib/kubelet/pods/19e166bd-4138-4d3d-9e2e-cbcca93e18f6/volumes/kubernetes.io~projected/kube-api-access-8z2w4 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e166bd-4138-4d3d-9e2e-cbcca93e18f6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/19e166bd-4138-4d3d-9e2e-cbcca93e18f6/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d14a7ee-7798-4273-9e07-68f3ad95b363/volumes/kubernetes.io~projected/kube-api-access-7jfx5:{mountpoint:/var/lib/kubelet/pods/1d14a7ee-7798-4273-9e07-68f3ad95b363/volumes/kubernetes.io~projected/kube-api-access-7jfx5 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25ed0359-9b20-49e1-be46-5da77b7853d3/volumes/kubernetes.io~projected/kube-api-access-wdrkq:{mountpoint:/var/lib/kubelet/pods/25ed0359-9b20-49e1-be46-5da77b7853d3/volumes/kubernetes.io~projected/kube-api-access-wdrkq major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~projected/kube-api-access-4d6t8:{mountpoint:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~projected/kube-api-access-4d6t8 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/438dc60b-6968-41c4-b974-cff36e50ab18/volumes/kubernetes.io~projected/kube-api-access-ndmkz:{mountpoint:/var/lib/kubelet/pods/438dc60b-6968-41c4-b974-cff36e50ab18/volumes/kubernetes.io~projected/kube-api-access-ndmkz major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/438dc60b-6968-41c4-b974-cff36e50ab18/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/438dc60b-6968-41c4-b974-cff36e50ab18/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/506e21c2-5241-44f6-8c30-0581ebe68600/volumes/kubernetes.io~projected/kube-api-access-gsmhs:{mountpoint:/var/lib/kubelet/pods/506e21c2-5241-44f6-8c30-0581ebe68600/volumes/kubernetes.io~projected/kube-api-access-gsmhs major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~projected/kube-api-access-5tlxc:{mountpoint:/var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~projected/kube-api-access-5tlxc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/611a6955-050e-4d16-b301-aa943a2f980e/volumes/kubernetes.io~projected/kube-api-access-8jp4p:{mountpoint:/var/lib/kubelet/pods/611a6955-050e-4d16-b301-aa943a2f980e/volumes/kubernetes.io~projected/kube-api-access-8jp4p major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e8f7ea-8d81-4b80-a527-f6e228885382/volumes/kubernetes.io~projected/kube-api-access-hzqzl:{mountpoint:/var/lib/kubelet/pods/62e8f7ea-8d81-4b80-a527-f6e228885382/volumes/kubernetes.io~projected/kube-api-access-hzqzl major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e8f7ea-8d81-4b80-a527-f6e228885382/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/62e8f7ea-8d81-4b80-a527-f6e228885382/volumes/kubernetes.io~secret/cert major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6449c7f8-c8f4-4022-8384-a85070a6493c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6449c7f8-c8f4-4022-8384-a85070a6493c/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73d52e94-07b5-4a34-b8ec-74e17ad7d096/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/73d52e94-07b5-4a34-b8ec-74e17ad7d096/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e919b41-8ea7-4e4f-a087-b5889c997875/volumes/kubernetes.io~projected/kube-api-access-vtvx5:{mountpoint:/var/lib/kubelet/pods/7e919b41-8ea7-4e4f-a087-b5889c997875/volumes/kubernetes.io~projected/kube-api-access-vtvx5 major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89367e5b-7b56-4918-95f3-5ae7c2ad5293/volumes/kubernetes.io~projected/kube-api-access-c5rz5:{mountpoint:/var/lib/kubelet/pods/89367e5b-7b56-4918-95f3-5ae7c2ad5293/volumes/kubernetes.io~projected/kube-api-access-c5rz5 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd/volumes/kubernetes.io~projected/kube-api-access-p6qvk:{mountpoint:/var/lib/kubelet/pods/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd/volumes/kubernetes.io~projected/kube-api-access-p6qvk major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~projected/kube-api-access-n6wwj:{mountpoint:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~projected/kube-api-access-n6wwj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~projected/kube-api-access-4h5hl:{mountpoint:/var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~projected/kube-api-access-4h5hl major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae3835ee-72cf-4adc-a485-b9c55d8601ce/volumes/kubernetes.io~projected/kube-api-access-zgg94:{mountpoint:/var/lib/kubelet/pods/ae3835ee-72cf-4adc-a485-b9c55d8601ce/volumes/kubernetes.io~projected/kube-api-access-zgg94 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ebff0e-2c89-4345-8577-9e296f299461/volumes/kubernetes.io~projected/kube-api-access-468nc:{mountpoint:/var/lib/kubelet/pods/c2ebff0e-2c89-4345-8577-9e296f299461/volumes/kubernetes.io~projected/kube-api-access-468nc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5d3993c-5c24-4912-9113-f64e2ca954b3/volumes/kubernetes.io~projected/kube-api-access-22rlh:{mountpoint:/var/lib/kubelet/pods/c5d3993c-5c24-4912-9113-f64e2ca954b3/volumes/kubernetes.io~projected/kube-api-access-22rlh major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce7abe8e-c9a4-4128-a744-0a2cd073c535/volumes/kubernetes.io~projected/kube-api-access-7ggvl:{mountpoint:/var/lib/kubelet/pods/ce7abe8e-c9a4-4128-a744-0a2cd073c535/volumes/kubernetes.io~projected/kube-api-access-7ggvl major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d38b2552-896f-498f-9719-5aa7c76561a2/volumes/kubernetes.io~projected/kube-api-access-h9r2w:{mountpoint:/var/lib/kubelet/pods/d38b2552-896f-498f-9719-5aa7c76561a2/volumes/kubernetes.io~projected/kube-api-access-h9r2w major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d86dbbbd-8799-4701-9fa5-831d1388c9d7/volumes/kubernetes.io~projected/kube-api-access-ljn92:{mountpoint:/var/lib/kubelet/pods/d86dbbbd-8799-4701-9fa5-831d1388c9d7/volumes/kubernetes.io~projected/kube-api-access-ljn92 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de009ad0-3b12-421f-a8bf-edfc92fd6b63/volumes/kubernetes.io~projected/kube-api-access-kpn4d:{mountpoint:/var/lib/kubelet/pods/de009ad0-3b12-421f-a8bf-edfc92fd6b63/volumes/kubernetes.io~projected/kube-api-access-kpn4d major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e38a56d4-3eca-4c4d-b7f9-fbd80403e692/volumes/kubernetes.io~projected/kube-api-access-74xth:{mountpoint:/var/lib/kubelet/pods/e38a56d4-3eca-4c4d-b7f9-fbd80403e692/volumes/kubernetes.io~projected/kube-api-access-74xth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/volumes/kubernetes.io~projected/kube-api-access-bbrkz:{mountpoint:/var/lib/kubelet/pods/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/volumes/kubernetes.io~projected/kube-api-access-bbrkz major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f25ae8a9-a299-4fdd-a752-6589f9f9b93a/volumes/kubernetes.io~projected/kube-api-access-64lrk:{mountpoint:/var/lib/kubelet/pods/f25ae8a9-a299-4fdd-a752-6589f9f9b93a/volumes/kubernetes.io~projected/kube-api-access-64lrk major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f68e237e-b990-4625-ba3a-a02cdb285bd8/volumes/kubernetes.io~projected/kube-api-access-z9g8j:{mountpoint:/var/lib/kubelet/pods/f68e237e-b990-4625-ba3a-a02cdb285bd8/volumes/kubernetes.io~projected/kube-api-access-z9g8j major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f86f89f2-60bc-44f1-b885-e33cc92bcd9f/volumes/kubernetes.io~projected/kube-api-access-8vt4f:{mountpoint:/var/lib/kubelet/pods/f86f89f2-60bc-44f1-b885-e33cc92bcd9f/volumes/kubernetes.io~projected/kube-api-access-8vt4f major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9db17c4-a168-4dfb-a89a-88933fd6413b/volumes/kubernetes.io~projected/kube-api-access-b4qzn:{mountpoint:/var/lib/kubelet/pods/f9db17c4-a168-4dfb-a89a-88933fd6413b/volumes/kubernetes.io~projected/kube-api-access-b4qzn major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d/volumes/kubernetes.io~projected/kube-api-access-c6k26:{mountpoint:/var/lib/kubelet/pods/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d/volumes/kubernetes.io~projected/kube-api-access-c6k26 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/67411fe40d156ae592045297aa3bd98bafe3acbfa27cd006dfe0071f0d9aefd1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/168864b54e94c45779c0f0bd29c88742d35619391b4bccb6fa433332eea08077/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b3fd88e9213bce79d200ceda85695fad775a3d11d8110a02383aa0d5b62329c3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/de1f3dcc4932927b7db9ba0b6ff5fdece1144f93e806b427e91452cd0cefec71/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a18daa48f58f8c18e89663782bd0382a8ed5cb89a0b60463e674e783e41f1141/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/30a3ec5f4dcae21ea1800957edf4117f44d17f2413b0e8deffa8ef88637e0485/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fdf9aa1458c3d2044f15e9b9c3b4c88afb64b5d2509836e81d310d4a711e992a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4766c81ccb2d825b82a092a75650c1357d4b443d0ab3021e7d938a900cfba074/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/666e80cd87ea95112b80779933bd427f618e9d7f3685baa727ba61e0db71d86b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6d78698b960c6be97f58e3171fbe08130b337320ea997b7a0846ece1cb4f7684/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8754df5b6adb835c5f8b92c37a79110767be724f5160c51e31247ad0e2cb3a2b/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/b433a4c95eb135429c08cb671c62c4d80ca0a999c76049f581dfc2361ce039ea/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cf6af37c02d8b737e62fab8831e328117d0d82678f030927353fbc08db1a0a9c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/be584c6e919163af915ffabfdf7427d06eedfb441d1b188341e599033554aac8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/268492839ce5a2519836f80ccac575904945320b361583a7b879bcba3634e96e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/76730e5f432f5f0e0a29aa1f8a245d1cfb3cb57008f795b4d6f5c77be15ea899/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8734b22a8255df7299a402f8eb6430b113b2aa9503ea2f688d8f03ab6565f96f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ebb9f71a52566721b3650f4ce84e50b847fb9b895276732312a6891e9c135cb2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/21dc9ad54907d1a613c9a6ad000fbc3177bec518a716cb26a8e75ac4fe8538a7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/88323db80f87307f3b6d1bac3cd84abc5d86332448d7698d849deded7846e684/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/a5f8458ec790abc5a0cf2b37767cb737bf4a7aa524556335c189a2315b954df8/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/fcd9371a661def6f3a8d832d70c530c882919c8390bf196638a3cd936de81425/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ce38e029fdc43e2c6acf5fc2615b6765bb01b13b8a85d5a119fb7fd19fc7b211/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/eadfdad951d03a1b23e8ff3cba4279b56a35501c02aeaff2eacecbb6c83e16ec/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/cf847ab2903994fc7037c767d1ec79b20cbc14434043acc3720ceb097f1c702c/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a0c11d44b0b963c60b1d5a33e600c31c8d5dec039581d45ee119d812d01a57a5/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/62a7d0dea6ebd52b504ae276f0a4d9e35f276c51f7f9cf51b48bb79c70628336/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/88ddd2773a0ac68741788b04ddb471d5056bc37ffb258b1f1ea7e2f3d10ba767/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/291f40d7726a8b92ff9bb976791cbb942cc212247bc338621172a5d4104045e5/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/363a125c5811b842b78a0396048204a9e5209cda31d877936dc7002bde0fbe70/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/9c91680e66bc599509abf9a4d45fd63da5f18c73d3fda35a5ac8d7f3153ee7c2/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/5fa01161e9189a89da8078314c3cd34d2a437b353a871a3defa15b59bb298f0e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/ce6fccebf0f57b20d6060d8c5e5ca376ca85d3bb2483e0348a52b4d060cfa9f4/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/2d684b528871ecb6005431feeced7aedb6eeafa1882fcd9f0219f5fe7e38e536/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/520269cf79cb787716a73b20ab3312731211e902027ef0c30c5ff1e93a3c075b/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/180de723c29a0979916c0b58194c683eb865c5d403c2c9d2ed4db177ba726742/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/8f7e6cf350f257c68ede1796a3b8f47a2cb9c1383b7c1756580f5b86c2df7841/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/1a55cd5fd84197a5dc4f35a931bba469cca86db6899a1ce155f1c6fe9f71db39/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/dbacb4f62614b10eb0186fff8f0f66f242c328a65552cf4218bb2a0b5a7e5719/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/40807da369848cc8c0fbae05b1b0408d4aa7151f9e0338ee40caad79fab2a5f4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/0167ec9776d49c92d75541d1c7168aba0c73621e78e8b32f3e1a8d441378b681/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/b72ac43a8477e5bf5f1db2114b97b2217c1886fe68948a61803d9bb13099d1e5/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/c7697f11d718c5f72973d0534cfe9a10333c27eb6d2d8c0a2ca60b3c84b38367/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/2e0cd245a439e4f87a281f028bbbe9146edf948a45105e5bd673518bfe2ddece/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/e4f3607cb2270be744f9866677acba9f81633eb87506d2cc5b41d8b07010121d/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/07f9fdfb987edaea365185f31368c127ba9cbb4589c770b8081ccd405ecb8774/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/8eebdeb2413540b0c35b71a2a33cb96db3b9082bd81b10c6c4ed29213fb4cec0/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/1ed2c14c6fcc00e7189029fc5e625dff170fc46753b1546c254f2a6c4ce62798/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/f8b67f7ba7541c55d6cf97d52b61e28ceb84d24bca9a046e0038f0dba1f7046c/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/2525e8c79b245cbf86f1a456e5dbb8bf56ef3909e58c2b7c3965e4db6c43d548/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/faf270aa0f456ebb004aaf8085a9169e99bf9d42aaed165127dfa3875fdae788/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/7eb090f077678eaa30612cfa95b2d0b8aea605120aa1eba1ab3140e11e80f4ba/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/4cc7f8f3f217454c3fc7e03fd0695fbeb5cf15df26f40a45bde532d0561ca6b3/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e740904a2c8b7808e1bbfdac403fbd1be6d5f0dbf4b1accb688d8b59b0f19710/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b1bcd1e8c33c02b841dbdabe8158ad87d5b69706cadeaec290fa2d8345f105a0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bab0c92a26cbee3a1bf9733e2cf6f57a9e8bfc21995fab6e3b9c99faa2e5ffd9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7e6c2324951b938c653a65b8f9616c991984d06470c5d975b8bf25b8639a6aa5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/32150e2103a024e95ad835078747146c64dd016c5cebabb3a08bf060343f388c/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ad511e91e3cb7292e7377d261d292ee38513952f2e953bf709fe01f77eae01f1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/d11cfd2959a360a2ca2430002cb19f1f5fc426f0deff1d37a336d78669f8b97a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6ed9ed82517934e6f0bd1e2439052b9a7db0950921c395b9d0c22b8670397dc3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/594d5c08aef179bbeba312daeb12c57de4c327843de8e55915eecd47d46b8948/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.998260 5055 manager.go:217] Machine: {Timestamp:2026-11-02 08:05:52.997762852 +0000 UTC m=+0.088383262 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100700 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c491031534c948e4e908ad82939a SystemUUID:ec20c491-0315-34c9-48e4-e908ad82939a BootID:7822187e-7314-44a0-b29d-c9def9634be1 Filesystems:[{Device:/var/lib/kubelet/pods/163e10f5-9315-4873-82b8-f830ff3135ac/volumes/kubernetes.io~projected/kube-api-access-9wkgx DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d/volumes/kubernetes.io~projected/kube-api-access-c6k26 DeviceMajor:0 DeviceMinor:361 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19e166bd-4138-4d3d-9e2e-cbcca93e18f6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/volumes/kubernetes.io~projected/kube-api-access-bbrkz DeviceMajor:0 DeviceMinor:227 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/de009ad0-3b12-421f-a8bf-edfc92fd6b63/volumes/kubernetes.io~projected/kube-api-access-kpn4d DeviceMajor:0 DeviceMinor:235 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/743ba67b5b0a41850028cb577a28d661c82e7426b8015b29c2e1733cded5fe3a/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f25ae8a9-a299-4fdd-a752-6589f9f9b93a/volumes/kubernetes.io~projected/kube-api-access-64lrk DeviceMajor:0 DeviceMinor:333 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05387920-8354-45bf-85e3-513462f484bf/volumes/kubernetes.io~projected/kube-api-access-dx6v6 DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e259ae1b109b638368deb3e54233f3774a9fa6cdaaa7a8eb23598ec0fdfdc64/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a052ffacbceb4ca15f6062ea67e341ba273fe4e1f5c5e135697d2beb5b5de0f1/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28136e9f743218292c0db79142b89153fb9e9fd19cec3df87a7d6121b58ea14c/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d2b9350af88d53ab48b0276f1653d30ed4af5e42f16fe50e32c3ecc467af79a/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/05387920-8354-45bf-85e3-513462f484bf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9db17c4-a168-4dfb-a89a-88933fd6413b/volumes/kubernetes.io~projected/kube-api-access-b4qzn DeviceMajor:0 DeviceMinor:234 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59cc5b1c16790930fa00f6928731d48dba521e4c002d55bda32507b173da6070/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25ed0359-9b20-49e1-be46-5da77b7853d3/volumes/kubernetes.io~projected/kube-api-access-wdrkq DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6449c7f8-c8f4-4022-8384-a85070a6493c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b29e4623a782350ba861fb6a1d362d6cb0afd40669c084d8785877feaf999b76/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c5d3993c-5c24-4912-9113-f64e2ca954b3/volumes/kubernetes.io~projected/kube-api-access-22rlh DeviceMajor:0 DeviceMinor:322 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~projected/kube-api-access-4h5hl DeviceMajor:0 DeviceMinor:53 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/89367e5b-7b56-4918-95f3-5ae7c2ad5293/volumes/kubernetes.io~projected/kube-api-access-c5rz5 DeviceMajor:0 DeviceMinor:52 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/753c1f27aa9cad87476436e9fa54660e31c1768e6079fea41b1efda5d16c774e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01ff3fed532fa7f60e66ad823804744d3abd1188531dcaae269de869444c916b/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volumes/kubernetes.io~projected/kube-api-access-tznrq DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2058ac1dfa1098d746ac969bcaf0d27ef177bce954adf7195debaf39d8c7613c/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0450940acceeca8d093ef89b301e53736c9b95b9438b271b7c0d3c1e21d126a5/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f9e331835f2cb40a98334b76676f86d816350174c4d4d3b3a2b1a28a8a78d3d/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ebff0e-2c89-4345-8577-9e296f299461/volumes/kubernetes.io~projected/kube-api-access-468nc DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~projected/kube-api-access-n6wwj DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~projected/kube-api-access-5tlxc DeviceMajor:0 DeviceMinor:223 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d86dbbbd-8799-4701-9fa5-831d1388c9d7/volumes/kubernetes.io~projected/kube-api-access-ljn92 DeviceMajor:0 DeviceMinor:327 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd/volumes/kubernetes.io~projected/kube-api-access-p6qvk DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1362e604911bbd10a1d5e887f6e2f1d4a9c5718e2427331854db3ebe93388e6b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19e166bd-4138-4d3d-9e2e-cbcca93e18f6/volumes/kubernetes.io~projected/kube-api-access-8z2w4 DeviceMajor:0 DeviceMinor:210 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e4901765efffb04a44d283eb4adb43224a2b5d9230623edaaea08d2452cb141/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~projected/kube-api-access-4d6t8 DeviceMajor:0 DeviceMinor:228 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7e919b41-8ea7-4e4f-a087-b5889c997875/volumes/kubernetes.io~projected/kube-api-access-vtvx5 DeviceMajor:0 DeviceMinor:332 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e8580088302156463bb2ff1ceea68f1357864464765dab3c1c6f221e732c388/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce7abe8e-c9a4-4128-a744-0a2cd073c535/volumes/kubernetes.io~projected/kube-api-access-7ggvl DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:212 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6045c48b-d8e9-4c92-8201-3ccc48a38a91/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/438dc60b-6968-41c4-b974-cff36e50ab18/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5ab6cc9581c8878f5deaf23b302fcb37f64b44ab51e40c51ac5556e7d21269b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/611a6955-050e-4d16-b301-aa943a2f980e/volumes/kubernetes.io~projected/kube-api-access-8jp4p DeviceMajor:0 DeviceMinor:226 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e2f19e389120f3090ed6cc3c5882ed547f5bf02f944a269eafc3b75473956d4/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d10977d65827ad609e13e9874a76e5b51f29f6a72840be4dd6d5c0111a4240ad/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1d14a7ee-7798-4273-9e07-68f3ad95b363/volumes/kubernetes.io~projected/kube-api-access-7jfx5 DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7279dc60e0dfdc85e2624f4fd15d9eac6d526f5f2d4a03dc172a5d4330b7a85/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/506e21c2-5241-44f6-8c30-0581ebe68600/volumes/kubernetes.io~projected/kube-api-access-gsmhs DeviceMajor:0 DeviceMinor:54 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a531717ff81f95e96cf8633d4680741d193e247ce909e6968c627b43dae08ac/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73d52e94-07b5-4a34-b8ec-74e17ad7d096/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f767997e303d4c0256036f17df351ef31dc9e5bc58687efa3cd2e90b80c9d21e/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d38b2552-896f-498f-9719-5aa7c76561a2/volumes/kubernetes.io~projected/kube-api-access-h9r2w DeviceMajor:0 DeviceMinor:51 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dc7cc27c935e88f357d524644f03c0e0863c93cce912f4c50aac96c9d57f711/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aeb28d1e4a4fbd6635375f22bad6cc4bdf9e967f881cb34ffb1ddef36fd0b728/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e38a56d4-3eca-4c4d-b7f9-fbd80403e692/volumes/kubernetes.io~projected/kube-api-access-74xth DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/62e8f7ea-8d81-4b80-a527-f6e228885382/volumes/kubernetes.io~projected/kube-api-access-hzqzl DeviceMajor:0 DeviceMinor:233 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/62e8f7ea-8d81-4b80-a527-f6e228885382/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:230 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f68e237e-b990-4625-ba3a-a02cdb285bd8/volumes/kubernetes.io~projected/kube-api-access-z9g8j DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae3835ee-72cf-4adc-a485-b9c55d8601ce/volumes/kubernetes.io~projected/kube-api-access-zgg94 DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/438dc60b-6968-41c4-b974-cff36e50ab18/volumes/kubernetes.io~projected/kube-api-access-ndmkz DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f86f89f2-60bc-44f1-b885-e33cc92bcd9f/volumes/kubernetes.io~projected/kube-api-access-8vt4f DeviceMajor:0 DeviceMinor:229 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01ff3fed532fa7f MacAddress:12:cf:2b:29:16:bd Speed:10000 Mtu:8901} {Name:0450940acceeca8 MacAddress:12:c3:9e:ac:35:fc Speed:10000 Mtu:8901} {Name:0e4901765efffb0 MacAddress:52:f8:8c:58:7f:39 Speed:10000 Mtu:8901} {Name:1362e604911bbd1 MacAddress:92:78:ad:9f:2d:11 Speed:10000 Mtu:8901} {Name:28136e9f7432182 MacAddress:6e:af:f8:9c:19:26 Speed:10000 Mtu:8901} {Name:2f9e331835f2cb4 MacAddress:62:bd:a6:33:72:60 Speed:10000 Mtu:8901} {Name:59cc5b1c1679093 MacAddress:fe:76:0f:f2:06:05 Speed:10000 Mtu:8901} {Name:5a531717ff81f95 MacAddress:b2:74:c4:d8:79:01 Speed:10000 Mtu:8901} {Name:743ba67b5b0a418 MacAddress:ca:92:17:39:16:f3 Speed:10000 Mtu:8901} {Name:7e259ae1b109b63 MacAddress:3e:0e:1c:2e:3b:cd Speed:10000 Mtu:8901} {Name:8e2f19e389120f3 MacAddress:e2:80:be:2b:1f:0f Speed:10000 Mtu:8901} {Name:8e8580088302156 MacAddress:d6:3d:a3:b2:07:bd Speed:10000 Mtu:8901} {Name:a052ffacbceb4ca MacAddress:d2:ca:55:73:b4:ef Speed:10000 Mtu:8901} {Name:b29e4623a782350 MacAddress:22:3d:cb:5d:83:80 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d7:bc:18:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:72:aa:ea:42:a0:d5 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:d7:bc:18:c1 Speed:0 Mtu:9001} {Name:f767997e303d4c0 MacAddress:4a:98:a9:06:8a:53 Speed:10000 Mtu:8901} {Name:fdbcc81a21f7409 MacAddress:b6:d2:be:ba:8a:98 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:26:bb:1a:de:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.998640 5055 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.998686 5055 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.998979 5055 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999004 5055 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-247.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999170 5055 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999178 5055 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999195 5055 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999215 5055 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999319 5055 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999479 5055 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999598 5055 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999633 5055 kubelet.go:493] "Attempting to sync node with API server" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999650 5055 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999660 5055 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999669 5055 kubelet.go:405] "Adding apiserver pod source" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:52.999682 5055 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.000819 5055 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.000833 5055 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:05:53.006153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.001458 5055 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 08:05:53.009587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.009562 5055 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 08:05:53.009587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.009585 5055 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:05:53.009729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.009702 5055 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 08:05:53.009912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.009900 5055 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 08:05:53.009963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.009930 5055 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 08:05:53.010049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010039 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 08:05:53.010105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010056 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 08:05:53.010105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010062 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 08:05:53.010105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010074 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 08:05:53.010105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010083 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 08:05:53.010105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010092 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 08:05:53.010105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010101 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010109 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010117 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010123 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010132 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010155 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010183 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 08:05:53.010383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.010192 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 08:05:53.013870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.013845 5055 server.go:1263] "Started kubelet" Nov 02 08:05:53.014027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.013977 5055 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 08:05:53.015067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.014289 5055 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 08:05:53.015067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.014371 5055 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 08:05:53.015067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.014811 5055 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 08:05:53.014440 ip-10-0-139-247 systemd[1]: Started Kubernetes Kubelet. Nov 02 08:05:53.016656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.015899 5055 server.go:317] "Adding debug handlers to kubelet server" Nov 02 08:05:53.019114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.019093 5055 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 08:05:53.019284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.019263 5055 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 08:05:53.020444 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.020387 5055 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 08:00:12 +0000 UTC" deadline="2028-06-18 22:59:27.391027472 +0000 UTC" Nov 02 08:05:53.020530 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.020500 5055 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" Nov 02 08:05:53.020596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.020542 5055 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 08:05:53.020596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.020551 5055 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 08:05:53.020796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.020778 5055 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.020445 5055 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14270h53m34.370586688s" Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024610 5055 factory.go:153] Registering CRI-O factory Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024628 5055 factory.go:223] Registration of the crio container factory successfully Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024696 5055 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024705 5055 factory.go:55] Registering systemd factory Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024713 5055 factory.go:223] Registration of the systemd container factory successfully Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024731 5055 factory.go:103] Registering Raw factory Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.024745 5055 manager.go:1199] Started watching for new ooms in manager Nov 02 08:05:53.026565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.025219 5055 manager.go:319] Starting recovery of all containers Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.028870 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.028942 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86" volumeName="kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-config" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.028967 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86" volumeName="kubernetes.io/projected/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-kube-api-access-tznrq" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.028982 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" volumeName="kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-trusted-ca-bundle" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029000 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029014 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a0e92c0-8b2e-439a-b72c-db5193d1c6bd" volumeName="kubernetes.io/projected/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd-kube-api-access-p6qvk" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029027 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86" volumeName="kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-env-overrides" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029046 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" volumeName="kubernetes.io/secret/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-serving-cert" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029121 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d14a7ee-7798-4273-9e07-68f3ad95b363" volumeName="kubernetes.io/projected/1d14a7ee-7798-4273-9e07-68f3ad95b363-kube-api-access-7jfx5" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029157 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="611a6955-050e-4d16-b301-aa943a2f980e" volumeName="kubernetes.io/projected/611a6955-050e-4d16-b301-aa943a2f980e-kube-api-access-8jp4p" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029171 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e8f7ea-8d81-4b80-a527-f6e228885382" volumeName="kubernetes.io/secret/62e8f7ea-8d81-4b80-a527-f6e228885382-cert" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029188 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6449c7f8-c8f4-4022-8384-a85070a6493c" volumeName="kubernetes.io/configmap/6449c7f8-c8f4-4022-8384-a85070a6493c-konnectivity-ca" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029206 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" volumeName="kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-snapshots" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029218 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029243 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029257 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029275 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86dbbbd-8799-4701-9fa5-831d1388c9d7" volumeName="kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029290 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e38a56d4-3eca-4c4d-b7f9-fbd80403e692" volumeName="kubernetes.io/projected/e38a56d4-3eca-4c4d-b7f9-fbd80403e692-kube-api-access-74xth" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029308 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86f89f2-60bc-44f1-b885-e33cc92bcd9f" volumeName="kubernetes.io/projected/f86f89f2-60bc-44f1-b885-e33cc92bcd9f-kube-api-access-8vt4f" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029320 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="163e10f5-9315-4873-82b8-f830ff3135ac" volumeName="kubernetes.io/configmap/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-config" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029333 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6045c48b-d8e9-4c92-8201-3ccc48a38a91" volumeName="kubernetes.io/projected/6045c48b-d8e9-4c92-8201-3ccc48a38a91-kube-api-access-5tlxc" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029350 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e8f7ea-8d81-4b80-a527-f6e228885382" volumeName="kubernetes.io/projected/62e8f7ea-8d81-4b80-a527-f6e228885382-kube-api-access-hzqzl" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029363 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73d52e94-07b5-4a34-b8ec-74e17ad7d096" volumeName="kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029380 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89367e5b-7b56-4918-95f3-5ae7c2ad5293" volumeName="kubernetes.io/projected/89367e5b-7b56-4918-95f3-5ae7c2ad5293-kube-api-access-c5rz5" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029394 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc3fb78-91ec-45b5-a9e2-a7f1101c8a04" volumeName="kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-tmp" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029406 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38b2552-896f-498f-9719-5aa7c76561a2" volumeName="kubernetes.io/configmap/d38b2552-896f-498f-9719-5aa7c76561a2-iptables-alerter-script" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029423 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a3cb18-3e9a-4452-88bb-a8c0515db6fd" volumeName="kubernetes.io/secret/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-serving-cert" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029435 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6045c48b-d8e9-4c92-8201-3ccc48a38a91" volumeName="kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-default-certificate" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029462 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ebff0e-2c89-4345-8577-9e296f299461" volumeName="kubernetes.io/projected/c2ebff0e-2c89-4345-8577-9e296f299461-kube-api-access-468nc" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029476 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86dbbbd-8799-4701-9fa5-831d1388c9d7" volumeName="kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029492 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a3cb18-3e9a-4452-88bb-a8c0515db6fd" volumeName="kubernetes.io/projected/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-kube-api-access-bbrkz" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029505 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05387920-8354-45bf-85e3-513462f484bf" volumeName="kubernetes.io/configmap/05387920-8354-45bf-85e3-513462f484bf-config" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029523 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029537 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029554 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029567 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506e21c2-5241-44f6-8c30-0581ebe68600" volumeName="kubernetes.io/empty-dir/506e21c2-5241-44f6-8c30-0581ebe68600-tmp-dir" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029580 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a3cb18-3e9a-4452-88bb-a8c0515db6fd" volumeName="kubernetes.io/configmap/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-config" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029623 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029639 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029658 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="438dc60b-6968-41c4-b974-cff36e50ab18" volumeName="kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-config" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029672 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="163e10f5-9315-4873-82b8-f830ff3135ac" volumeName="kubernetes.io/projected/163e10f5-9315-4873-82b8-f830ff3135ac-kube-api-access-9wkgx" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029686 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ed0359-9b20-49e1-be46-5da77b7853d3" volumeName="kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029703 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6449c7f8-c8f4-4022-8384-a85070a6493c" volumeName="kubernetes.io/secret/6449c7f8-c8f4-4022-8384-a85070a6493c-agent-certs" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029716 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e919b41-8ea7-4e4f-a087-b5889c997875" volumeName="kubernetes.io/projected/7e919b41-8ea7-4e4f-a087-b5889c997875-kube-api-access-vtvx5" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029732 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86dbbbd-8799-4701-9fa5-831d1388c9d7" volumeName="kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029747 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f68e237e-b990-4625-ba3a-a02cdb285bd8" volumeName="kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029760 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029779 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" volumeName="kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-service-ca-bundle" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029795 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ed0359-9b20-49e1-be46-5da77b7853d3" volumeName="kubernetes.io/projected/25ed0359-9b20-49e1-be46-5da77b7853d3-kube-api-access-wdrkq" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029813 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89367e5b-7b56-4918-95f3-5ae7c2ad5293" volumeName="kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-daemon-config" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029827 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d3993c-5c24-4912-9113-f64e2ca954b3" volumeName="kubernetes.io/empty-dir/c5d3993c-5c24-4912-9113-f64e2ca954b3-data-volume" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029839 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6491032-ad9f-4d34-98b8-6fb3f03e4cd9" volumeName="kubernetes.io/configmap/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-nginx-conf" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029859 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de009ad0-3b12-421f-a8bf-edfc92fd6b63" volumeName="kubernetes.io/empty-dir/de009ad0-3b12-421f-a8bf-edfc92fd6b63-tmp-dir" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029873 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff50b9eb-13e4-4e99-8dcd-b14db45dff7d" volumeName="kubernetes.io/projected/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-kube-api-access-c6k26" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029890 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05387920-8354-45bf-85e3-513462f484bf" volumeName="kubernetes.io/secret/05387920-8354-45bf-85e3-513462f484bf-serving-cert" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029903 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" volumeName="kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029934 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25ae8a9-a299-4fdd-a752-6589f9f9b93a" volumeName="kubernetes.io/projected/f25ae8a9-a299-4fdd-a752-6589f9f9b93a-kube-api-access-64lrk" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029952 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86" volumeName="kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-script-lib" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029964 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="438dc60b-6968-41c4-b974-cff36e50ab18" volumeName="kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-trusted-ca" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029977 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc3fb78-91ec-45b5-a9e2-a7f1101c8a04" volumeName="kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-tuned" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.029993 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d3993c-5c24-4912-9113-f64e2ca954b3" volumeName="kubernetes.io/projected/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-api-access-22rlh" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030006 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff50b9eb-13e4-4e99-8dcd-b14db45dff7d" volumeName="kubernetes.io/configmap/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-cabundle" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030023 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" volumeName="kubernetes.io/projected/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-kube-api-access-8z2w4" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030035 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030048 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="438dc60b-6968-41c4-b974-cff36e50ab18" volumeName="kubernetes.io/secret/438dc60b-6968-41c4-b974-cff36e50ab18-serving-cert" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030076 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7abe8e-c9a4-4128-a744-0a2cd073c535" volumeName="kubernetes.io/projected/ce7abe8e-c9a4-4128-a744-0a2cd073c535-kube-api-access-7ggvl" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030090 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de009ad0-3b12-421f-a8bf-edfc92fd6b63" volumeName="kubernetes.io/configmap/de009ad0-3b12-421f-a8bf-edfc92fd6b63-config-volume" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030107 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de009ad0-3b12-421f-a8bf-edfc92fd6b63" volumeName="kubernetes.io/projected/de009ad0-3b12-421f-a8bf-edfc92fd6b63-kube-api-access-kpn4d" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030119 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9db17c4-a168-4dfb-a89a-88933fd6413b" volumeName="kubernetes.io/projected/f9db17c4-a168-4dfb-a89a-88933fd6413b-kube-api-access-b4qzn" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030132 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1c4943-feda-467b-9d73-33f613a74060" volumeName="kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030195 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86" volumeName="kubernetes.io/secret/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030209 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ed0359-9b20-49e1-be46-5da77b7853d3" volumeName="kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030228 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506e21c2-5241-44f6-8c30-0581ebe68600" volumeName="kubernetes.io/projected/506e21c2-5241-44f6-8c30-0581ebe68600-kube-api-access-gsmhs" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030240 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89367e5b-7b56-4918-95f3-5ae7c2ad5293" volumeName="kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cni-binary-copy" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030257 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae3835ee-72cf-4adc-a485-b9c55d8601ce" volumeName="kubernetes.io/projected/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kube-api-access-zgg94" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030269 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ebff0e-2c89-4345-8577-9e296f299461" volumeName="kubernetes.io/configmap/c2ebff0e-2c89-4345-8577-9e296f299461-serviceca" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030281 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" volumeName="kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-tmp" seLinuxMountContext="" Nov 02 08:05:53.030239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030299 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ed0359-9b20-49e1-be46-5da77b7853d3" volumeName="kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-binary-copy" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030311 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="438dc60b-6968-41c4-b974-cff36e50ab18" volumeName="kubernetes.io/projected/438dc60b-6968-41c4-b974-cff36e50ab18-kube-api-access-ndmkz" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030346 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6045c48b-d8e9-4c92-8201-3ccc48a38a91" volumeName="kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-stats-auth" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030362 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="611a6955-050e-4d16-b301-aa943a2f980e" volumeName="kubernetes.io/configmap/611a6955-050e-4d16-b301-aa943a2f980e-telemetry-config" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030378 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38b2552-896f-498f-9719-5aa7c76561a2" volumeName="kubernetes.io/projected/d38b2552-896f-498f-9719-5aa7c76561a2-kube-api-access-h9r2w" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030400 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff50b9eb-13e4-4e99-8dcd-b14db45dff7d" volumeName="kubernetes.io/secret/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-key" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030413 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05387920-8354-45bf-85e3-513462f484bf" volumeName="kubernetes.io/projected/05387920-8354-45bf-85e3-513462f484bf-kube-api-access-dx6v6" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030430 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc3fb78-91ec-45b5-a9e2-a7f1101c8a04" volumeName="kubernetes.io/projected/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-kube-api-access-4h5hl" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030443 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86dbbbd-8799-4701-9fa5-831d1388c9d7" volumeName="kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle" seLinuxMountContext="" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030455 5055 reconstruct.go:97] "Volume reconstruction finished" Nov 02 08:05:53.037913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.030462 5055 reconciler.go:29] "Reconciler: start to sync state" Nov 02 08:05:53.050135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.049818 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:05:53.051024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.050985 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:05:53.079878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.079286 5055 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 08:05:53.085229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.083650 5055 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 08:05:53.085229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.083673 5055 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 08:05:53.085229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.083694 5055 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 08:05:53.085229 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.083733 5055 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 08:05:53.105205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.105120 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:05:53.105205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.105175 5055 generic.go:356] "Generic (PLEG): container finished" podID="438dc60b-6968-41c4-b974-cff36e50ab18" containerID="96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8" exitCode=255 Nov 02 08:05:53.119778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.119738 5055 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="8f5d8fdadbb1f2431736e8f6356cd8a45809cc942c43569276ea607738736a36" exitCode=0 Nov 02 08:05:53.119778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.119776 5055 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="86fe0a1b52e28d2658a92c7a99bad14a3e8e85a438297f9fdb367514be722599" exitCode=0 Nov 02 08:05:53.119778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.119785 5055 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="570abe045677af91fe6ef8075531c888f4bdac100d4c21735fd0c0d8d925f39b" exitCode=0 Nov 02 08:05:53.119994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.119792 5055 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="2642834de699f4969375c606d44131b3ed330b56cb112b510258b75441e38641" exitCode=0 Nov 02 08:05:53.119994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.119799 5055 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="69889d2c64483954688b1352216bbbf6fbf22ac5078f6f8eac36056d4a031751" exitCode=0 Nov 02 08:05:53.119994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.119807 5055 generic.go:356] "Generic (PLEG): container finished" podID="25ed0359-9b20-49e1-be46-5da77b7853d3" containerID="f72c70d8524fb09ca9e26ebe27bbcbe1d9bd656fb7aeefe594196ee11ae7a1e0" exitCode=0 Nov 02 08:05:53.120638 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.120615 5055 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" Nov 02 08:05:53.139968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.139930 5055 generic.go:356] "Generic (PLEG): container finished" podID="104fe0f8525d94a2e03d5f4707bfa4d0" containerID="cc2b1b24f28cb996e2d8395f9b1346cd2aa2d6ab2f05cb199f916e1ef0162f75" exitCode=0 Nov 02 08:05:53.149252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.149216 5055 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:53.149938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.149915 5055 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:53.176526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.176500 5055 manager.go:324] Recovery completed Nov 02 08:05:53.183833 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.183807 5055 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 08:05:53.196037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.196011 5055 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 08:05:53.197362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.197347 5055 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:53.198536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.198518 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:53.198616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.198547 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:53.198616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.198562 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:53.201635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201615 5055 cpu_manager.go:226] "Starting" policy="none" Nov 02 08:05:53.201635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201630 5055 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 08:05:53.201743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201647 5055 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 08:05:53.201743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201643 5055 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:53.201821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201781 5055 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 08:05:53.201821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201794 5055 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 08:05:53.201821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201810 5055 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 08:05:53.201821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201815 5055 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 08:05:53.201821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201820 5055 policy_none.go:50] "Start" Nov 02 08:05:53.201997 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201829 5055 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 08:05:53.201997 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.201838 5055 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 08:05:53.202060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.202016 5055 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 08:05:53.202060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.202028 5055 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 08:05:53.202060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.202037 5055 policy_none.go:44] "Start" Nov 02 08:05:53.202666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.202649 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:53.202726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.202687 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:53.202726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.202701 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:53.207061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.207035 5055 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 08:05:53.209704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.209689 5055 manager.go:341] "Starting Device Plugin manager" Nov 02 08:05:53.209812 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.209717 5055 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 08:05:53.209812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.209725 5055 server.go:87] "Starting device plugin registration server" Nov 02 08:05:53.210504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210489 5055 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 08:05:53.210563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210508 5055 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 08:05:53.210691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210681 5055 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 08:05:53.210782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210758 5055 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 08:05:53.210828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210789 5055 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 08:05:53.210828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210794 5055 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 08:05:53.210916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.210854 5055 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T08:05:53.210780317Z","UUID":"4fab5a94-ddb3-4918-9198-49ae7a096e5b","Handler":null,"Name":"","Endpoint":""} Nov 02 08:05:53.213984 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.213407 5055 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 08:05:53.213984 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.213466 5055 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-247.us-west-2.compute.internal\" not found" Nov 02 08:05:53.213984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.213518 5055 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 08:05:53.213984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.213555 5055 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 08:05:53.241471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.241434 5055 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:53.243694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.243668 5055 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:53.310663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.310621 5055 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:53.311894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.311873 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:53.311982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.311912 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:53.311982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.311928 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-247.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:53.312073 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.311987 5055 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.384257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.384194 5055 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal"] Nov 02 08:05:53.384883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.384841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"34d28ca7a91586006c95d24bde05028b","Type":"ContainerStarted","Data":"42310aa681edcd3e7d39ccab8ec10589e393c41af35d14666a4779fcdbdae45e"} Nov 02 08:05:53.384883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.384886 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"34d28ca7a91586006c95d24bde05028b","Type":"ContainerStarted","Data":"57fb42b9fbf45564b6b524a59dc077901b5ec502a77880366a26144d8072aabe"} Nov 02 08:05:53.385041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.384954 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"104fe0f8525d94a2e03d5f4707bfa4d0","Type":"ContainerStarted","Data":"dc61793d9e3592958db8157fc033375f7086a294b52861a33c1e8a3a80a866e2"} Nov 02 08:05:53.385041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.384969 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"104fe0f8525d94a2e03d5f4707bfa4d0","Type":"ContainerDied","Data":"cc2b1b24f28cb996e2d8395f9b1346cd2aa2d6ab2f05cb199f916e1ef0162f75"} Nov 02 08:05:53.385041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.384981 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" event={"ID":"104fe0f8525d94a2e03d5f4707bfa4d0","Type":"ContainerStarted","Data":"c11e0b548eaf4385ae0475edd3bb8619068b70517e2ad937266496a331e41c67"} Nov 02 08:05:53.385041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.385013 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.385235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.385076 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.413688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.413660 5055 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.413822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.413730 5055 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.446946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.446913 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:53.447096 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.446980 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.447701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.447689 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:53.447823 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:53.447723 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.497394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.497360 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.497561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.497400 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.497561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.497419 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34d28ca7a91586006c95d24bde05028b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"34d28ca7a91586006c95d24bde05028b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.598195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.598133 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.598348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.598204 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.598348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.598213 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.598348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.598224 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34d28ca7a91586006c95d24bde05028b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"34d28ca7a91586006c95d24bde05028b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.598348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.598260 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34d28ca7a91586006c95d24bde05028b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"34d28ca7a91586006c95d24bde05028b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:53.598348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:53.598307 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104fe0f8525d94a2e03d5f4707bfa4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" (UID: \"104fe0f8525d94a2e03d5f4707bfa4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:54.001906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.001865 5055 apiserver.go:52] "Watching apiserver" Nov 02 08:05:54.145906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.145873 5055 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 08:05:54.146653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.146624 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-8d2bg","kube-system/konnectivity-agent-hlm8n","kube-system/global-pull-secret-syncer-2phd8","openshift-network-operator/iptables-alerter-vc95q","openshift-dns/dns-default-q49jt","openshift-service-ca/service-ca-5fffc97bdf-lw557","kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd","openshift-insights/insights-runtime-extractor-48rm2","openshift-image-registry/image-registry-595bc5b6bc-js2wl","openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f","openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm","openshift-console-operator/console-operator-7cd56b8dbc-tspm9","openshift-cluster-node-tuning-operator/tuned-nldh5","openshift-multus/network-metrics-daemon-trgc8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8","openshift-insights/periodic-gathering-t29pm-98s5b","openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs","openshift-multus/multus-g4xwk","kube-system/kas-connection-checker-685895c7c9-cbw8w","openshift-ovn-kubernetes/ovnkube-node-2sz6m","openshift-network-console/networking-console-plugin-86c478bb88-wtksb","kube-system/kas-connection-checker-685895c7c9-86sd2","kube-system/kas-connection-checker-685895c7c9-qmp75","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd","openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx","openshift-image-registry/node-ca-s52jj","openshift-multus/multus-additional-cni-plugins-phvgd","openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk","openshift-image-registry/image-registry-59f6c46786-fzsst","openshift-insights/insights-operator-7c4986d449-fr2k5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-svhfk","openshift-dns/node-resolver-m29jr","openshift-ingress/router-default-d5df879b9-lxqd6"] Nov 02 08:05:54.147707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.147688 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.148480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.148459 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.150030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.150011 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.150937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.150921 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.151882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.151863 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.155416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.155363 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:54.160838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.160818 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.161956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.161940 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.162932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.162918 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.177769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.177748 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:54.177901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.177889 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:54.196285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.196261 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.196369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.196294 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.197985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.197968 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 08:05:54.198029 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.197990 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 08:05:54.198093 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198080 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 08:05:54.198296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198283 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 08:05:54.198450 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198438 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.198526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198509 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 08:05:54.198579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198554 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.198579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198557 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198577 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198544 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198615 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198617 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198618 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198639 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198649 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.198683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198641 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198727 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198748 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198781 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198790 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198817 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198831 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198833 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198877 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 08:05:54.199000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.198878 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.199452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.199120 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.200234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.200198 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 08:05:54.200373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.200354 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 08:05:54.200654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.200632 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.223044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.223022 5055 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 08:05:54.242851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.242830 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.244876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.244860 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 08:05:54.246540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246370 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 08:05:54.246676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246618 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 08:05:54.246741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246685 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 08:05:54.246741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246698 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.246741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246713 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 08:05:54.246888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246761 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.246888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246789 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 08:05:54.246888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246860 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 08:05:54.247027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.246959 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.247124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247106 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 08:05:54.247199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247186 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 08:05:54.247251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247189 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 08:05:54.247251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247219 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.247251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247227 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.247251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247234 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.247251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247232 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247222 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247306 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247316 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247325 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247316 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4gc2j\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247350 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247354 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247352 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247370 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247412 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247417 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247423 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247420 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247441 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247457 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.247482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247481 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247509 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247521 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247572 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rwld\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247593 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zgn8x\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247597 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247633 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9ztjt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247671 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247687 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4v7vr\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247691 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247707 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t9rxp\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247690 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247734 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247778 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5nd8v\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247798 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247804 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247813 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kvw72\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247821 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247848 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247856 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247869 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.247923 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248028 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nct\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248107 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-c87zx\"" Nov 02 08:05:54.248127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248132 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248169 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwch9\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.248191 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248271 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghx8m\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248275 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248335 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qmmzq\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248385 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-shlrq\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248507 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248530 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248565 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bgzdd\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248622 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248663 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zrnf9\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248695 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248710 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248757 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2bt7\"" Nov 02 08:05:54.248818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248826 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jw25f\"" Nov 02 08:05:54.249270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248873 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9tsd7\"" Nov 02 08:05:54.249270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248934 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 08:05:54.249270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248948 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w8vvg\"" Nov 02 08:05:54.249270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.248983 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 08:05:54.256048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.256007 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:54.256098 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.256048 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-247.us-west-2.compute.internal" Nov 02 08:05:54.301403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301373 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38b2552-896f-498f-9719-5aa7c76561a2-host-slash\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.301509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301411 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfx5\" (UniqueName: \"kubernetes.io/projected/1d14a7ee-7798-4273-9e07-68f3ad95b363-kube-api-access-7jfx5\") pod \"kas-connection-checker-685895c7c9-cbw8w\" (UID: \"1d14a7ee-7798-4273-9e07-68f3ad95b363\") " pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:54.301509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301431 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.301509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301448 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.301607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301512 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-log-socket\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.301607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301539 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-modprobe-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.301607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301562 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-kubelet-config\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.301695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301641 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpn4d\" (UniqueName: \"kubernetes.io/projected/de009ad0-3b12-421f-a8bf-edfc92fd6b63-kube-api-access-kpn4d\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.301695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301675 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/611a6955-050e-4d16-b301-aa943a2f980e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.301807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301791 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggvl\" (UniqueName: \"kubernetes.io/projected/ce7abe8e-c9a4-4128-a744-0a2cd073c535-kube-api-access-7ggvl\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:54.301908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301892 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt4f\" (UniqueName: \"kubernetes.io/projected/f86f89f2-60bc-44f1-b885-e33cc92bcd9f-kube-api-access-8vt4f\") pod \"volume-data-source-validator-58c9f5f46d-d6dk8\" (UID: \"f86f89f2-60bc-44f1-b885-e33cc92bcd9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:54.301941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301927 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.301970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301955 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.302006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.301981 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e8f7ea-8d81-4b80-a527-f6e228885382-cert\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:54.302043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302014 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.302084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302041 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:54.302084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302038 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/611a6955-050e-4d16-b301-aa943a2f980e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.302173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302127 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-cabundle\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:54.302218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302182 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jp4p\" (UniqueName: \"kubernetes.io/projected/611a6955-050e-4d16-b301-aa943a2f980e-kube-api-access-8jp4p\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.302218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302208 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.302295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302235 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.302295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302254 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e8f7ea-8d81-4b80-a527-f6e228885382-cert\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:54.302295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302287 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.302442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302306 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.302442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302327 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.302442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302330 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-cabundle\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:54.302442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302416 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302449 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302452 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302478 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302494 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302521 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302556 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-conf-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302577 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.302634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302624 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302621 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rlh\" (UniqueName: \"kubernetes.io/projected/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-api-access-22rlh\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302680 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xth\" (UniqueName: \"kubernetes.io/projected/e38a56d4-3eca-4c4d-b7f9-fbd80403e692-kube-api-access-74xth\") pod \"kas-connection-checker-685895c7c9-86sd2\" (UID: \"e38a56d4-3eca-4c4d-b7f9-fbd80403e692\") " pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302741 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6449c7f8-c8f4-4022-8384-a85070a6493c-konnectivity-ca\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302802 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-device-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302864 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-script-lib\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302895 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302930 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5hl\" (UniqueName: \"kubernetes.io/projected/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-kube-api-access-4h5hl\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302960 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmkz\" (UniqueName: \"kubernetes.io/projected/438dc60b-6968-41c4-b974-cff36e50ab18-kube-api-access-ndmkz\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.302992 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d3993c-5c24-4912-9113-f64e2ca954b3-data-volume\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303009 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6449c7f8-c8f4-4022-8384-a85070a6493c-konnectivity-ca\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303021 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-default-certificate\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.303049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303052 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/25ed0359-9b20-49e1-be46-5da77b7853d3-kube-api-access-wdrkq\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303082 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d3993c-5c24-4912-9113-f64e2ca954b3-data-volume\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303085 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-slash\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303129 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6449c7f8-c8f4-4022-8384-a85070a6493c-agent-certs\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303178 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303193 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303272 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-script-lib\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303288 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-default-certificate\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303309 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvx5\" (UniqueName: \"kubernetes.io/projected/7e919b41-8ea7-4e4f-a087-b5889c997875-kube-api-access-vtvx5\") pod \"migrator-66df5d496b-tczzs\" (UID: \"7e919b41-8ea7-4e4f-a087-b5889c997875\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303339 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303376 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438dc60b-6968-41c4-b974-cff36e50ab18-serving-cert\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r2w\" (UniqueName: \"kubernetes.io/projected/d38b2552-896f-498f-9719-5aa7c76561a2-kube-api-access-h9r2w\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303435 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-stats-auth\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303489 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-system-cni-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303492 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73d52e94-07b5-4a34-b8ec-74e17ad7d096-original-pull-secret\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303507 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6449c7f8-c8f4-4022-8384-a85070a6493c-agent-certs\") pod \"konnectivity-agent-hlm8n\" (UID: \"6449c7f8-c8f4-4022-8384-a85070a6493c\") " pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303516 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.303575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303559 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-multus\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303587 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303633 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-bin\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303656 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303683 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-stats-auth\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303724 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de009ad0-3b12-421f-a8bf-edfc92fd6b63-config-volume\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303736 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ed0359-9b20-49e1-be46-5da77b7853d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303757 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de009ad0-3b12-421f-a8bf-edfc92fd6b63-tmp-dir\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303817 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z2w4\" (UniqueName: \"kubernetes.io/projected/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-kube-api-access-8z2w4\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303827 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de009ad0-3b12-421f-a8bf-edfc92fd6b63-tmp-dir\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303848 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438dc60b-6968-41c4-b974-cff36e50ab18-serving-cert\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303855 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-sys-fs\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303885 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgg94\" (UniqueName: \"kubernetes.io/projected/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kube-api-access-zgg94\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303905 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de009ad0-3b12-421f-a8bf-edfc92fd6b63-config-volume\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303929 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303949 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-hostroot\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303965 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-serving-cert\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.303981 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304015 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrkz\" (UniqueName: \"kubernetes.io/projected/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-kube-api-access-bbrkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304040 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysconfig\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304058 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-registration-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304073 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304120 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-serving-cert\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304130 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304158 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304172 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-key\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304192 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468nc\" (UniqueName: \"kubernetes.io/projected/c2ebff0e-2c89-4345-8577-9e296f299461-kube-api-access-468nc\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.304307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304260 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304332 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznrq\" (UniqueName: \"kubernetes.io/projected/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-kube-api-access-tznrq\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304360 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304375 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304418 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-kubelet\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304467 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-signing-key\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304502 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-var-lib-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304593 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-etc-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304631 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304675 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-sys\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304733 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-socket-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304777 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304793 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304871 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304873 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304903 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-system-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304938 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-daemon-config\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304972 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.304975 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305039 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-service-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305097 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqzl\" (UniqueName: \"kubernetes.io/projected/62e8f7ea-8d81-4b80-a527-f6e228885382-kube-api-access-hzqzl\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305129 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305173 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cnibin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305199 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-bin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305248 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-run\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305247 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305269 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-daemon-config\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305296 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-lib-modules\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305335 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305336 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-trusted-ca\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.305512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305455 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506e21c2-5241-44f6-8c30-0581ebe68600-tmp-dir\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305542 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305570 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305588 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-trusted-ca\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305626 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k26\" (UniqueName: \"kubernetes.io/projected/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-kube-api-access-c6k26\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305634 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-service-ca-bundle\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305655 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506e21c2-5241-44f6-8c30-0581ebe68600-tmp-dir\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305676 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305734 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-systemd-units\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305770 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305807 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-var-lib-kubelet\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305860 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-cnibin\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305895 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-etc-kubernetes\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305951 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-systemd\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305987 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.305988 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-tmp\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306016 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306024 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306037 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-tmp\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306065 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306095 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306121 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-conf\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306182 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkgx\" (UniqueName: \"kubernetes.io/projected/163e10f5-9315-4873-82b8-f830ff3135ac-kube-api-access-9wkgx\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306211 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-tuned\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306320 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qzn\" (UniqueName: \"kubernetes.io/projected/f9db17c4-a168-4dfb-a89a-88933fd6413b-kube-api-access-b4qzn\") pod \"network-check-source-6f78d59f99-2ngtk\" (UID: \"f9db17c4-a168-4dfb-a89a-88933fd6413b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306350 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-tuned\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306353 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6v6\" (UniqueName: \"kubernetes.io/projected/05387920-8354-45bf-85e3-513462f484bf-kube-api-access-dx6v6\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306429 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-kubelet\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306465 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-tmp\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.306493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306499 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-netns\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306522 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-tmp\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306530 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05387920-8354-45bf-85e3-513462f484bf-serving-cert\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306558 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38b2552-896f-498f-9719-5aa7c76561a2-iptables-alerter-script\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306626 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lrk\" (UniqueName: \"kubernetes.io/projected/f25ae8a9-a299-4fdd-a752-6589f9f9b93a-kube-api-access-64lrk\") pod \"migrator-66df5d496b-k9qrm\" (UID: \"f25ae8a9-a299-4fdd-a752-6589f9f9b93a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306695 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-ovn\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306715 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05387920-8354-45bf-85e3-513462f484bf-serving-cert\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306729 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-node-log\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306767 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38b2552-896f-498f-9719-5aa7c76561a2-iptables-alerter-script\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306774 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmhs\" (UniqueName: \"kubernetes.io/projected/506e21c2-5241-44f6-8c30-0581ebe68600-kube-api-access-gsmhs\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306796 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306812 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-k8s-cni-cncf-io\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306835 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-config\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306872 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306922 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-netns\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306952 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.306977 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-config\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307004 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307033 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cni-binary-copy\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307070 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307102 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-host\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307134 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05387920-8354-45bf-85e3-513462f484bf-config\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rz5\" (UniqueName: \"kubernetes.io/projected/89367e5b-7b56-4918-95f3-5ae7c2ad5293-kube-api-access-c5rz5\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307246 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307268 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307284 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-socket-dir-parent\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307316 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-snapshots\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307331 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2ebff0e-2c89-4345-8577-9e296f299461-host\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307347 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5d3993c-5c24-4912-9113-f64e2ca954b3-crio-socket\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307364 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlxc\" (UniqueName: \"kubernetes.io/projected/6045c48b-d8e9-4c92-8201-3ccc48a38a91-kube-api-access-5tlxc\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307380 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307385 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05387920-8354-45bf-85e3-513462f484bf-config\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307397 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovn-node-metrics-cert\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307414 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-kubernetes\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307432 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qvk\" (UniqueName: \"kubernetes.io/projected/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd-kube-api-access-p6qvk\") pod \"kas-connection-checker-685895c7c9-qmp75\" (UID: \"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd\") " pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:54.307439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307431 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307477 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307511 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2ebff0e-2c89-4345-8577-9e296f299461-serviceca\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307520 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307541 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-env-overrides\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307555 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovnkube-config\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307561 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cni-binary-copy\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307567 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307573 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-snapshots\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307579 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/438dc60b-6968-41c4-b974-cff36e50ab18-config\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307630 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-multus-certs\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307679 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-ovn-node-metrics-cert\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307750 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-systemd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307791 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-netd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307815 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2ebff0e-2c89-4345-8577-9e296f299461-serviceca\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/506e21c2-5241-44f6-8c30-0581ebe68600-hosts-file\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307848 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307866 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-dbus\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307864 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-env-overrides\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.308671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.307980 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.354613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.354575 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpn4d\" (UniqueName: \"kubernetes.io/projected/de009ad0-3b12-421f-a8bf-edfc92fd6b63-kube-api-access-kpn4d\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.354735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.354628 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggvl\" (UniqueName: \"kubernetes.io/projected/ce7abe8e-c9a4-4128-a744-0a2cd073c535-kube-api-access-7ggvl\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:54.354735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.354679 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfx5\" (UniqueName: \"kubernetes.io/projected/1d14a7ee-7798-4273-9e07-68f3ad95b363-kube-api-access-7jfx5\") pod \"kas-connection-checker-685895c7c9-cbw8w\" (UID: \"1d14a7ee-7798-4273-9e07-68f3ad95b363\") " pod="kube-system/kas-connection-checker-685895c7c9-cbw8w" Nov 02 08:05:54.354892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.354870 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt4f\" (UniqueName: \"kubernetes.io/projected/f86f89f2-60bc-44f1-b885-e33cc92bcd9f-kube-api-access-8vt4f\") pod \"volume-data-source-validator-58c9f5f46d-d6dk8\" (UID: \"f86f89f2-60bc-44f1-b885-e33cc92bcd9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d6dk8" Nov 02 08:05:54.355054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.355038 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jp4p\" (UniqueName: \"kubernetes.io/projected/611a6955-050e-4d16-b301-aa943a2f980e-kube-api-access-8jp4p\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.355357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.355336 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5hl\" (UniqueName: \"kubernetes.io/projected/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-kube-api-access-4h5hl\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.355765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.355745 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.355836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.355783 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rlh\" (UniqueName: \"kubernetes.io/projected/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-api-access-22rlh\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.356991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.356964 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r2w\" (UniqueName: \"kubernetes.io/projected/d38b2552-896f-498f-9719-5aa7c76561a2-kube-api-access-h9r2w\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.357085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.357047 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xth\" (UniqueName: \"kubernetes.io/projected/e38a56d4-3eca-4c4d-b7f9-fbd80403e692-kube-api-access-74xth\") pod \"kas-connection-checker-685895c7c9-86sd2\" (UID: \"e38a56d4-3eca-4c4d-b7f9-fbd80403e692\") " pod="kube-system/kas-connection-checker-685895c7c9-86sd2" Nov 02 08:05:54.358433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.358410 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmkz\" (UniqueName: \"kubernetes.io/projected/438dc60b-6968-41c4-b974-cff36e50ab18-kube-api-access-ndmkz\") pod \"console-operator-7cd56b8dbc-tspm9\" (UID: \"438dc60b-6968-41c4-b974-cff36e50ab18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:54.358541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.358411 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrkz\" (UniqueName: \"kubernetes.io/projected/e3a3cb18-3e9a-4452-88bb-a8c0515db6fd-kube-api-access-bbrkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd\" (UID: \"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" Nov 02 08:05:54.358606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.358560 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.359168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.359060 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznrq\" (UniqueName: \"kubernetes.io/projected/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-kube-api-access-tznrq\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.359262 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.359195 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgg94\" (UniqueName: \"kubernetes.io/projected/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kube-api-access-zgg94\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.359319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.359284 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvx5\" (UniqueName: \"kubernetes.io/projected/7e919b41-8ea7-4e4f-a087-b5889c997875-kube-api-access-vtvx5\") pod \"migrator-66df5d496b-tczzs\" (UID: \"7e919b41-8ea7-4e4f-a087-b5889c997875\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tczzs" Nov 02 08:05:54.360166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.359359 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z2w4\" (UniqueName: \"kubernetes.io/projected/19e166bd-4138-4d3d-9e2e-cbcca93e18f6-kube-api-access-8z2w4\") pod \"insights-operator-7c4986d449-fr2k5\" (UID: \"19e166bd-4138-4d3d-9e2e-cbcca93e18f6\") " pod="openshift-insights/insights-operator-7c4986d449-fr2k5" Nov 02 08:05:54.360166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.359630 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/25ed0359-9b20-49e1-be46-5da77b7853d3-kube-api-access-wdrkq\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.363856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.363495 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqzl\" (UniqueName: \"kubernetes.io/projected/62e8f7ea-8d81-4b80-a527-f6e228885382-kube-api-access-hzqzl\") pod \"ingress-canary-svhfk\" (UID: \"62e8f7ea-8d81-4b80-a527-f6e228885382\") " pod="openshift-ingress-canary/ingress-canary-svhfk" Nov 02 08:05:54.363856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.363619 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-468nc\" (UniqueName: \"kubernetes.io/projected/c2ebff0e-2c89-4345-8577-9e296f299461-kube-api-access-468nc\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.363856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.363810 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k26\" (UniqueName: \"kubernetes.io/projected/ff50b9eb-13e4-4e99-8dcd-b14db45dff7d-kube-api-access-c6k26\") pod \"service-ca-5fffc97bdf-lw557\" (UID: \"ff50b9eb-13e4-4e99-8dcd-b14db45dff7d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lw557" Nov 02 08:05:54.363856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.363840 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qzn\" (UniqueName: \"kubernetes.io/projected/f9db17c4-a168-4dfb-a89a-88933fd6413b-kube-api-access-b4qzn\") pod \"network-check-source-6f78d59f99-2ngtk\" (UID: \"f9db17c4-a168-4dfb-a89a-88933fd6413b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2ngtk" Nov 02 08:05:54.363856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.363855 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6v6\" (UniqueName: \"kubernetes.io/projected/05387920-8354-45bf-85e3-513462f484bf-kube-api-access-dx6v6\") pod \"service-ca-operator-544d749c8c-hhvjx\" (UID: \"05387920-8354-45bf-85e3-513462f484bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" Nov 02 08:05:54.364048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.363810 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lrk\" (UniqueName: \"kubernetes.io/projected/f25ae8a9-a299-4fdd-a752-6589f9f9b93a-kube-api-access-64lrk\") pod \"migrator-66df5d496b-k9qrm\" (UID: \"f25ae8a9-a299-4fdd-a752-6589f9f9b93a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k9qrm" Nov 02 08:05:54.364210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.364124 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkgx\" (UniqueName: \"kubernetes.io/projected/163e10f5-9315-4873-82b8-f830ff3135ac-kube-api-access-9wkgx\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.364210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.364131 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g8j\" (UniqueName: \"kubernetes.io/projected/f68e237e-b990-4625-ba3a-a02cdb285bd8-kube-api-access-z9g8j\") pod \"network-check-target-8d2bg\" (UID: \"f68e237e-b990-4625-ba3a-a02cdb285bd8\") " pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:05:54.364993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.364975 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") pod \"periodic-gathering-t29pm-98s5b\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:05:54.365067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.365047 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rz5\" (UniqueName: \"kubernetes.io/projected/89367e5b-7b56-4918-95f3-5ae7c2ad5293-kube-api-access-c5rz5\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.365132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.365068 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.365285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.365157 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.365435 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.365416 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmhs\" (UniqueName: \"kubernetes.io/projected/506e21c2-5241-44f6-8c30-0581ebe68600-kube-api-access-gsmhs\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.365602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.365586 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qvk\" (UniqueName: \"kubernetes.io/projected/8a0e92c0-8b2e-439a-b72c-db5193d1c6bd-kube-api-access-p6qvk\") pod \"kas-connection-checker-685895c7c9-qmp75\" (UID: \"8a0e92c0-8b2e-439a-b72c-db5193d1c6bd\") " pod="kube-system/kas-connection-checker-685895c7c9-qmp75" Nov 02 08:05:54.365761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.365746 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlxc\" (UniqueName: \"kubernetes.io/projected/6045c48b-d8e9-4c92-8201-3ccc48a38a91-kube-api-access-5tlxc\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.408581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408557 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-kubelet\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408587 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-netns\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408603 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-ovn\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408617 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-node-log\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408640 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408659 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-k8s-cni-cncf-io\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408661 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-netns\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408680 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-ovn\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408695 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-k8s-cni-cncf-io\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408683 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-kubelet\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.408711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408708 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-netns\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408731 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-node-log\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408733 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-netns\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408742 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408770 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-host\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.408785 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408796 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-socket-dir-parent\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408824 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2ebff0e-2c89-4345-8577-9e296f299461-host\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408849 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5d3993c-5c24-4912-9113-f64e2ca954b3-crio-socket\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408856 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-socket-dir-parent\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.408859 5055 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408864 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-host\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408890 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5d3993c-5c24-4912-9113-f64e2ca954b3-crio-socket\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408890 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2ebff0e-2c89-4345-8577-9e296f299461-host\") pod \"node-ca-s52jj\" (UID: \"c2ebff0e-2c89-4345-8577-9e296f299461\") " pod="openshift-image-registry/node-ca-s52jj" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.408870 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:54.908848778 +0000 UTC m=+1.999469199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.408986 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:54.908971385 +0000 UTC m=+1.999591795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.408997 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-kubernetes\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409016 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409033 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409055 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-multus-certs\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409058 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-kubernetes\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409070 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-systemd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409085 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-netd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.409104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409108 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/506e21c2-5241-44f6-8c30-0581ebe68600-hosts-file\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409158 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-dbus\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409178 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409185 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-run-multus-certs\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409193 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38b2552-896f-498f-9719-5aa7c76561a2-host-slash\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409223 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/506e21c2-5241-44f6-8c30-0581ebe68600-hosts-file\") pod \"node-resolver-m29jr\" (UID: \"506e21c2-5241-44f6-8c30-0581ebe68600\") " pod="openshift-dns/node-resolver-m29jr" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409227 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409233 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-dbus\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409186 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-systemd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409233 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38b2552-896f-498f-9719-5aa7c76561a2-host-slash\") pod \"iptables-alerter-vc95q\" (UID: \"d38b2552-896f-498f-9719-5aa7c76561a2\") " pod="openshift-network-operator/iptables-alerter-vc95q" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409168 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-netd\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409291 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409313 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-log-socket\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409332 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-modprobe-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409353 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409370 5055 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409395 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-kubelet-config\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409416 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409435 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409396 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-modprobe-d\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409377 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-log-socket\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409492 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73d52e94-07b5-4a34-b8ec-74e17ad7d096-kubelet-config\") pod \"global-pull-secret-syncer-2phd8\" (UID: \"73d52e94-07b5-4a34-b8ec-74e17ad7d096\") " pod="kube-system/global-pull-secret-syncer-2phd8" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409500 5055 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409527 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409535 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:54.909522847 +0000 UTC m=+2.000143280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : secret "metrics-daemon-secret" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409554 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409570 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-conf-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409586 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409605 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-device-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409619 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-multus-conf-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409658 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409670 5055 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409687 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-slash\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409718 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-system-cni-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409727 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:54.909712558 +0000 UTC m=+2.000332972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : configmap "kube-rbac-proxy" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409749 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-device-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409752 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-system-cni-dir\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409757 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-multus\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409797 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-multus\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409805 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409819 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409831 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409834 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-slash\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409830 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-bin\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409883 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-cni-bin\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409874 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-run-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.409902 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:54.909885908 +0000 UTC m=+2.000506311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409949 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409977 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-sys-fs\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.409999 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-hostroot\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410000 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410020 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410048 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysconfig\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410057 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-sys-fs\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-registration-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410107 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-registration-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410110 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.410090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410132 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-hostroot\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410165 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-kubelet\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410191 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-var-lib-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410216 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-etc-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410222 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysconfig\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410241 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-sys\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410246 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-kubelet\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410249 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410263 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-socket-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410275 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-etc-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410253 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6045c48b-d8e9-4c92-8201-3ccc48a38a91-service-ca-bundle\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410292 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-system-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410194 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410301 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-sys\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410316 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae3835ee-72cf-4adc-a485-b9c55d8601ce-socket-dir\") pod \"aws-ebs-csi-driver-node-467hd\" (UID: \"ae3835ee-72cf-4adc-a485-b9c55d8601ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-467hd" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410300 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-var-lib-openvswitch\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410330 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cnibin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410342 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-system-cni-dir\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410354 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-cnibin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410354 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-bin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410377 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-run\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410387 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-host-var-lib-cni-bin\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410391 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-lib-modules\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410427 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410443 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-run\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410453 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-systemd-units\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410462 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-lib-modules\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410481 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-var-lib-kubelet\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410499 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-systemd-units\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410505 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-cnibin\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.410514 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.410527 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410530 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-etc-kubernetes\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410544 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-var-lib-kubelet\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410548 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ed0359-9b20-49e1-be46-5da77b7853d3-cnibin\") pod \"multus-additional-cni-plugins-phvgd\" (UID: \"25ed0359-9b20-49e1-be46-5da77b7853d3\") " pod="openshift-multus/multus-additional-cni-plugins-phvgd" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410553 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-systemd\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.410563 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:54.910549938 +0000 UTC m=+2.001170334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410593 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410601 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89367e5b-7b56-4918-95f3-5ae7c2ad5293-etc-kubernetes\") pod \"multus-g4xwk\" (UID: \"89367e5b-7b56-4918-95f3-5ae7c2ad5293\") " pod="openshift-multus/multus-g4xwk" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410620 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410637 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-systemd\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410645 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-conf\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410664 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sz6m\" (UID: \"0b96f0e5-7abd-41ca-ae54-7c8f35d5cb86\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.410803 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abc3fb78-91ec-45b5-a9e2-a7f1101c8a04-etc-sysctl-conf\") pod \"tuned-nldh5\" (UID: \"abc3fb78-91ec-45b5-a9e2-a7f1101c8a04\") " pod="openshift-cluster-node-tuning-operator/tuned-nldh5" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.412018 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de009ad0-3b12-421f-a8bf-edfc92fd6b63-metrics-tls\") pod \"dns-default-q49jt\" (UID: \"de009ad0-3b12-421f-a8bf-edfc92fd6b63\") " pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.412068 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6491032-ad9f-4d34-98b8-6fb3f03e4cd9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wtksb\" (UID: \"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.412621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.412527 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/163e10f5-9315-4873-82b8-f830ff3135ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2n4f\" (UID: \"163e10f5-9315-4873-82b8-f830ff3135ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.413921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.412939 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/611a6955-050e-4d16-b301-aa943a2f980e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-srmdt\" (UID: \"611a6955-050e-4d16-b301-aa943a2f980e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.448773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.448740 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" Nov 02 08:05:54.450838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.450818 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" Nov 02 08:05:54.450905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.450894 5055 scope.go:122] "RemoveContainer" containerID="96431717907f2cf20463204bdcf7427e821be6c32e56d7c4c794d045fbb01aa8" Nov 02 08:05:54.452080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.452060 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" Nov 02 08:05:54.462729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.462709 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:54.646622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.644954 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-svhfk_62e8f7ea-8d81-4b80-a527-f6e228885382/serve-healthcheck-canary/0.log" Nov 02 08:05:54.844728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.844650 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-k9qrm_f25ae8a9-a299-4fdd-a752-6589f9f9b93a/migrator/0.log" Nov 02 08:05:54.887573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.887543 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f"] Nov 02 08:05:54.910846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.910772 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt"] Nov 02 08:05:54.914468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.914442 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:54.914573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.914499 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.914573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.914531 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:54.914666 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914588 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:54.914666 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914612 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:54.914666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.914613 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:54.914666 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914630 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.914688 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.914716 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914719 5055 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914772 5055 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914775 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:55.914761246 +0000 UTC m=+3.005381674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914824 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:55.914808858 +0000 UTC m=+3.005429258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : configmap "kube-rbac-proxy" not found Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914831 5055 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 08:05:54.914849 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914845 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:54.915292 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914839 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:55.914830482 +0000 UTC m=+3.005450882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:54.915292 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914859 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:54.915292 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914873 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:55.914863907 +0000 UTC m=+3.005484336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : secret "metrics-daemon-secret" not found Nov 02 08:05:54.915292 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914895 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:55.914886288 +0000 UTC m=+3.005506690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:54.915292 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:54.914910 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:55.914902691 +0000 UTC m=+3.005523103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:54.929695 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:54.929666 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611a6955_050e_4d16_b301_aa943a2f980e.slice/crio-2366f55966d3884d4d22404b2c2345c0b0025b20de362cd569e334af00ff6c75 WatchSource:0}: Error finding container 2366f55966d3884d4d22404b2c2345c0b0025b20de362cd569e334af00ff6c75: Status 404 returned error can't find the container with id 2366f55966d3884d4d22404b2c2345c0b0025b20de362cd569e334af00ff6c75 Nov 02 08:05:54.931631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.931613 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:05:54.937599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.937580 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q49jt"] Nov 02 08:05:54.940872 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:54.940851 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde009ad0_3b12_421f_a8bf_edfc92fd6b63.slice/crio-0f189c6dada6451c1c4dad609ac226739cb809a4afe1e715cce9ce168d58cebe WatchSource:0}: Error finding container 0f189c6dada6451c1c4dad609ac226739cb809a4afe1e715cce9ce168d58cebe: Status 404 returned error can't find the container with id 0f189c6dada6451c1c4dad609ac226739cb809a4afe1e715cce9ce168d58cebe Nov 02 08:05:54.959532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:54.959504 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wtksb"] Nov 02 08:05:54.962949 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:05:54.962923 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6491032_ad9f_4d34_98b8_6fb3f03e4cd9.slice/crio-27629846ca7166934cad18018b83be3dc726d07c44cdba6a18049be0beb2edbe WatchSource:0}: Error finding container 27629846ca7166934cad18018b83be3dc726d07c44cdba6a18049be0beb2edbe: Status 404 returned error can't find the container with id 27629846ca7166934cad18018b83be3dc726d07c44cdba6a18049be0beb2edbe Nov 02 08:05:55.044615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.044580 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tczzs_7e919b41-8ea7-4e4f-a087-b5889c997875/migrator/0.log" Nov 02 08:05:55.183687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.183619 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:05:55.183830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.183699 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" event={"ID":"438dc60b-6968-41c4-b974-cff36e50ab18","Type":"ContainerStarted","Data":"c88143a459ea8e65c5d33136e7d7b6f0dc13fc005b97be3e60e2839a8d3831f0"} Nov 02 08:05:55.188955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.185532 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q49jt" event={"ID":"de009ad0-3b12-421f-a8bf-edfc92fd6b63","Type":"ContainerStarted","Data":"0f189c6dada6451c1c4dad609ac226739cb809a4afe1e715cce9ce168d58cebe"} Nov 02 08:05:55.190539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.190509 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" event={"ID":"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9","Type":"ContainerStarted","Data":"27629846ca7166934cad18018b83be3dc726d07c44cdba6a18049be0beb2edbe"} Nov 02 08:05:55.191682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.191658 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" event={"ID":"611a6955-050e-4d16-b301-aa943a2f980e","Type":"ContainerStarted","Data":"2366f55966d3884d4d22404b2c2345c0b0025b20de362cd569e334af00ff6c75"} Nov 02 08:05:55.192711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.192687 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" event={"ID":"163e10f5-9315-4873-82b8-f830ff3135ac","Type":"ContainerStarted","Data":"6b53e07e574be6248a279effef2ceaa0072da603c22f7dc8f61496f2af781a50"} Nov 02 08:05:55.243120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.243087 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd_e3a3cb18-3e9a-4452-88bb-a8c0515db6fd/kube-storage-version-migrator-operator/0.log" Nov 02 08:05:55.511838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.511811 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:55.905179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.905075 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:55.908949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.908921 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tspm9" Nov 02 08:05:55.924747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.924714 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:55.924937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.924771 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:55.924937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.924812 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:55.924937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.924865 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:55.924937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.924899 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:55.924937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.924920 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:55.925113 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.925028 5055 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 08:05:55.925113 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.925076 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs podName:6045c48b-d8e9-4c92-8201-3ccc48a38a91 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.925063578 +0000 UTC m=+5.015683974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs") pod "router-default-d5df879b9-lxqd6" (UID: "6045c48b-d8e9-4c92-8201-3ccc48a38a91") : secret "router-metrics-certs-default" not found Nov 02 08:05:55.925500 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.925471 5055 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 08:05:55.925877 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.925535 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.925516843 +0000 UTC m=+5.016137258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : secret "metrics-daemon-secret" not found Nov 02 08:05:55.926177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:55.926134 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5d3993c-5c24-4912-9113-f64e2ca954b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:55.926263 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926241 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:55.926263 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926254 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:55.926341 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926296 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.926280938 +0000 UTC m=+5.016901337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:55.926378 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926366 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:55.926407 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926379 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:55.926439 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926417 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.9264053 +0000 UTC m=+5.017025699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:55.926519 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926506 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:55.926555 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:55.926547 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:57.926534725 +0000 UTC m=+5.017155123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:57.947801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:57.947708 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:05:57.947801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:57.947759 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:57.947805 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:57.947832 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:57.947882 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.947902 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.947985 5055 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948001 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.947977614 +0000 UTC m=+9.038598012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948037 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948052 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f6c46786-fzsst: secret "image-registry-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948059 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.948041718 +0000 UTC m=+9.038662147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : secret "metrics-daemon-secret" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.947993 5055 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948079 5055 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595bc5b6bc-js2wl: secret "image-registry-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948106 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls podName:2e1c4943-feda-467b-9d73-33f613a74060 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.948091998 +0000 UTC m=+9.038712408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls") pod "image-registry-59f6c46786-fzsst" (UID: "2e1c4943-feda-467b-9d73-33f613a74060") : secret "image-registry-tls" not found Nov 02 08:05:57.948377 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:05:57.948123 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls podName:95dd7a9d-509f-48cd-b87e-c50f688c4a9a nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.948114578 +0000 UTC m=+9.038734977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls") pod "image-registry-595bc5b6bc-js2wl" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a") : secret "image-registry-tls" not found Nov 02 08:05:57.950620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:57.950596 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6045c48b-d8e9-4c92-8201-3ccc48a38a91-metrics-certs\") pod \"router-default-d5df879b9-lxqd6\" (UID: \"6045c48b-d8e9-4c92-8201-3ccc48a38a91\") " pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:58.052196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.052162 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:05:58.206322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.206290 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wtksb" event={"ID":"d6491032-ad9f-4d34-98b8-6fb3f03e4cd9","Type":"ContainerStarted","Data":"3042e93000d7e321bfe293b17a449821319aaac947e15e854c399b484750e53f"} Nov 02 08:05:58.207723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.207696 5055 generic.go:356] "Generic (PLEG): container finished" podID="19e166bd-4138-4d3d-9e2e-cbcca93e18f6" containerID="9f6083b4f7eace095c0dbe458476bee0b1b9b4e10ca9d5f04eaa9d7a516a10bd" exitCode=0 Nov 02 08:05:58.207847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.207735 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" event={"ID":"19e166bd-4138-4d3d-9e2e-cbcca93e18f6","Type":"ContainerDied","Data":"9f6083b4f7eace095c0dbe458476bee0b1b9b4e10ca9d5f04eaa9d7a516a10bd"} Nov 02 08:05:58.207999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.207987 5055 scope.go:122] "RemoveContainer" containerID="9f6083b4f7eace095c0dbe458476bee0b1b9b4e10ca9d5f04eaa9d7a516a10bd" Nov 02 08:05:58.669943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.669686 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d5df879b9-lxqd6"] Nov 02 08:05:58.920947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.920869 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:05:58.921118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:58.921107 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:05:59.212974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.212943 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q49jt" event={"ID":"de009ad0-3b12-421f-a8bf-edfc92fd6b63","Type":"ContainerStarted","Data":"64b94b4bb657bf0f25492ebad3efec3e2bd78aac121c85d95358a681c7295f3f"} Nov 02 08:05:59.213445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.212992 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q49jt" event={"ID":"de009ad0-3b12-421f-a8bf-edfc92fd6b63","Type":"ContainerStarted","Data":"c9e244b6fba12f22c04f7004016deea9bd786509e7c500e22e03bb51e10bc6d3"} Nov 02 08:05:59.213445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.213041 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q49jt" Nov 02 08:05:59.214867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.214841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d5df879b9-lxqd6" event={"ID":"6045c48b-d8e9-4c92-8201-3ccc48a38a91","Type":"ContainerStarted","Data":"b6b4d7e91fdda5ad8fbdbc3636ce94093b87564e10e972cf1d471af0198e8aa7"} Nov 02 08:05:59.215006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.214884 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d5df879b9-lxqd6" event={"ID":"6045c48b-d8e9-4c92-8201-3ccc48a38a91","Type":"ContainerStarted","Data":"2621bc90b4af5e9f8397db14200ae1d2e3a06f21c9b1a48cf1f32107d71f5f6d"} Nov 02 08:05:59.216718 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.216690 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fr2k5" event={"ID":"19e166bd-4138-4d3d-9e2e-cbcca93e18f6","Type":"ContainerStarted","Data":"c46b176f8577f186935e5b444eed47d1bb65775d3634a18fb62abda1b9675ffb"} Nov 02 08:05:59.218321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.218290 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-srmdt" event={"ID":"611a6955-050e-4d16-b301-aa943a2f980e","Type":"ContainerStarted","Data":"2fab9f75718af8eead42561b87feca58a34ba31ebdd2e0bf5810fcd6de65e069"} Nov 02 08:05:59.220057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.220033 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" event={"ID":"163e10f5-9315-4873-82b8-f830ff3135ac","Type":"ContainerStarted","Data":"32a0af37c9b9e8d4e56c7ffad6cf36fc0d974c95c03326c9d46cbf8a23d5f22f"} Nov 02 08:05:59.220170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.220095 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2n4f" event={"ID":"163e10f5-9315-4873-82b8-f830ff3135ac","Type":"ContainerStarted","Data":"221f4f1687afcbff4eda389314df30ba039ac0085e420ef1fee73c653d28ffce"} Nov 02 08:05:59.539085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:05:59.538998 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sz6m" Nov 02 08:06:00.052994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:00.052954 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:06:00.055812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:00.055788 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:06:00.061900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:00.061878 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:06:01.228443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.228408 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:06:01.979227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.979187 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:06:01.979437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.979240 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:06:01.979437 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:01.979354 5055 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 08:06:01.979437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.979364 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:06:01.979437 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:01.979394 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 08:06:01.979437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.979430 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:01.979702 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:01.979467 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls podName:c5d3993c-5c24-4912-9113-f64e2ca954b3 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:09.979452269 +0000 UTC m=+17.070072666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48rm2" (UID: "c5d3993c-5c24-4912-9113-f64e2ca954b3") : secret "insights-runtime-extractor-tls" not found Nov 02 08:06:01.979702 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:01.979500 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs podName:ce7abe8e-c9a4-4128-a744-0a2cd073c535 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:09.979477155 +0000 UTC m=+17.070097563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs") pod "network-metrics-daemon-trgc8" (UID: "ce7abe8e-c9a4-4128-a744-0a2cd073c535") : secret "metrics-daemon-secret" not found Nov 02 08:06:01.981747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.981716 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"image-registry-595bc5b6bc-js2wl\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:06:01.981747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:01.981726 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"image-registry-59f6c46786-fzsst\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:02.249096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:02.249019 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:02.261242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:02.261204 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:06:02.719013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:02.718984 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f6c46786-fzsst"] Nov 02 08:06:02.721626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:02.721602 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-595bc5b6bc-js2wl"] Nov 02 08:06:02.722366 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:02.722345 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1c4943_feda_467b_9d73_33f613a74060.slice/crio-c3516146e61451e04517ba0c81c3fceb06027f4cfe3d44dfd8abb35704b54a20 WatchSource:0}: Error finding container c3516146e61451e04517ba0c81c3fceb06027f4cfe3d44dfd8abb35704b54a20: Status 404 returned error can't find the container with id c3516146e61451e04517ba0c81c3fceb06027f4cfe3d44dfd8abb35704b54a20 Nov 02 08:06:02.768218 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:02.768184 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dd7a9d_509f_48cd_b87e_c50f688c4a9a.slice/crio-5791b6f6782d4af46fb0d3dc545cc567eb14fd8bfb6a56348d81c6bd8bc9f99f WatchSource:0}: Error finding container 5791b6f6782d4af46fb0d3dc545cc567eb14fd8bfb6a56348d81c6bd8bc9f99f: Status 404 returned error can't find the container with id 5791b6f6782d4af46fb0d3dc545cc567eb14fd8bfb6a56348d81c6bd8bc9f99f Nov 02 08:06:03.235880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:03.235834 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" event={"ID":"2e1c4943-feda-467b-9d73-33f613a74060","Type":"ContainerStarted","Data":"8a191af5c09a99f5a9b6c120efb22925be90be6870e89afb5de0c0e7d70bbb26"} Nov 02 08:06:03.236077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:03.235902 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" event={"ID":"2e1c4943-feda-467b-9d73-33f613a74060","Type":"ContainerStarted","Data":"c3516146e61451e04517ba0c81c3fceb06027f4cfe3d44dfd8abb35704b54a20"} Nov 02 08:06:03.237530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:03.237507 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" event={"ID":"95dd7a9d-509f-48cd-b87e-c50f688c4a9a","Type":"ContainerStarted","Data":"4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4"} Nov 02 08:06:03.237641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:03.237544 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" event={"ID":"95dd7a9d-509f-48cd-b87e-c50f688c4a9a","Type":"ContainerStarted","Data":"5791b6f6782d4af46fb0d3dc545cc567eb14fd8bfb6a56348d81c6bd8bc9f99f"} Nov 02 08:06:04.242339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:04.242307 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:06:04.242854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:04.242308 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:06:07.080762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:07.080725 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:07.869348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:07.869312 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:06:07.869555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:07.869472 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:06:07.871186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:07.871160 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d5df879b9-lxqd6" Nov 02 08:06:09.226592 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:09.226560 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q49jt" Nov 02 08:06:10.050280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.050240 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:06:10.050456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.050327 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:06:10.052631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.052598 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5d3993c-5c24-4912-9113-f64e2ca954b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48rm2\" (UID: \"c5d3993c-5c24-4912-9113-f64e2ca954b3\") " pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:06:10.052750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.052653 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7abe8e-c9a4-4128-a744-0a2cd073c535-metrics-certs\") pod \"network-metrics-daemon-trgc8\" (UID: \"ce7abe8e-c9a4-4128-a744-0a2cd073c535\") " pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:06:10.056644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.056623 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trgc8" Nov 02 08:06:10.064109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.064093 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48rm2" Nov 02 08:06:10.418099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.418006 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-trgc8"] Nov 02 08:06:10.459091 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:10.459063 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7abe8e_c9a4_4128_a744_0a2cd073c535.slice/crio-9239f9dffd0a671eb4c5f4ceea6e1f71496de55a53bda4a641d931379089e050 WatchSource:0}: Error finding container 9239f9dffd0a671eb4c5f4ceea6e1f71496de55a53bda4a641d931379089e050: Status 404 returned error can't find the container with id 9239f9dffd0a671eb4c5f4ceea6e1f71496de55a53bda4a641d931379089e050 Nov 02 08:06:10.467465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:10.467397 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48rm2"] Nov 02 08:06:10.469677 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:10.469658 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d3993c_5c24_4912_9113_f64e2ca954b3.slice/crio-9ac1003ebb5286d507949766d682a93763b1eaa7351df173d9f06618ddb632ca WatchSource:0}: Error finding container 9ac1003ebb5286d507949766d682a93763b1eaa7351df173d9f06618ddb632ca: Status 404 returned error can't find the container with id 9ac1003ebb5286d507949766d682a93763b1eaa7351df173d9f06618ddb632ca Nov 02 08:06:11.269196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:11.269157 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trgc8" event={"ID":"ce7abe8e-c9a4-4128-a744-0a2cd073c535","Type":"ContainerStarted","Data":"9239f9dffd0a671eb4c5f4ceea6e1f71496de55a53bda4a641d931379089e050"} Nov 02 08:06:11.270601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:11.270574 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48rm2" event={"ID":"c5d3993c-5c24-4912-9113-f64e2ca954b3","Type":"ContainerStarted","Data":"343e739854329d9a991099047da495677f85d8deb846434c210967eea17a041b"} Nov 02 08:06:11.270703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:11.270624 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48rm2" event={"ID":"c5d3993c-5c24-4912-9113-f64e2ca954b3","Type":"ContainerStarted","Data":"9ac1003ebb5286d507949766d682a93763b1eaa7351df173d9f06618ddb632ca"} Nov 02 08:06:11.745225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:11.745195 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:06:12.131695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.131611 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6m7vz-27w65"] Nov 02 08:06:12.138598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.138569 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.169822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.169481 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.169822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.169532 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.169822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.169580 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.169822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.169609 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.177357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.177326 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf"] Nov 02 08:06:12.179912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.179891 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:12.184063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.184038 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-66qfc"] Nov 02 08:06:12.186827 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.186808 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.194076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.194050 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ffcbccbb7-ndg2f"] Nov 02 08:06:12.197184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.196577 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.224227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.224201 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59f6c46786-fzsst"] Nov 02 08:06:12.224989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.224966 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6m7vz-27w65"] Nov 02 08:06:12.228851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.228827 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vhhrt\"" Nov 02 08:06:12.228963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.228895 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 08:06:12.238761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.238739 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 08:06:12.238853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.238833 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 08:06:12.245028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.245007 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 08:06:12.245871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.245846 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.277993 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.280216 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64466fbc47-7k2d9"] Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.280553 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.280604 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.280657 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.280682 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwk22\" (UniqueName: \"kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.281120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.280761 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a7ed96f-845a-44ac-915e-ff9d26cf2a6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-jkvkf\" (UID: \"1a7ed96f-845a-44ac-915e-ff9d26cf2a6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:12.281661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.281639 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.281785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.281747 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282287 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282325 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e584897d-a9b9-424a-aba1-29a0dc184e3e-tmp\") pod \"downloads-6dd789c8b7-66qfc\" (UID: \"e584897d-a9b9-424a-aba1-29a0dc184e3e\") " pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282357 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfb4q\" (UniqueName: \"kubernetes.io/projected/e584897d-a9b9-424a-aba1-29a0dc184e3e-kube-api-access-vfb4q\") pod \"downloads-6dd789c8b7-66qfc\" (UID: \"e584897d-a9b9-424a-aba1-29a0dc184e3e\") " pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282397 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282426 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282455 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282484 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282517 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.282917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.282858 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.283379 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.283357 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.284222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.284200 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf"] Nov 02 08:06:12.284310 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.284245 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-66qfc"] Nov 02 08:06:12.284372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.284342 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.289028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.288999 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48rm2" event={"ID":"c5d3993c-5c24-4912-9113-f64e2ca954b3","Type":"ContainerStarted","Data":"893ff1c8967b962d651d747cd542ebfb7f33dcc2de51ad37022530ac2039e324"} Nov 02 08:06:12.312649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.312621 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffcbccbb7-ndg2f"] Nov 02 08:06:12.326746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.326717 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64466fbc47-7k2d9"] Nov 02 08:06:12.328857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.328833 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 08:06:12.328996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.328869 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-httw9\"" Nov 02 08:06:12.328996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.328907 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wxtll\"" Nov 02 08:06:12.328996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.328955 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 08:06:12.335909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.335876 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt\") pod \"periodic-gathering-6m7vz-27w65\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.383647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383553 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwk22\" (UniqueName: \"kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.383647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383619 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a7ed96f-845a-44ac-915e-ff9d26cf2a6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-jkvkf\" (UID: \"1a7ed96f-845a-44ac-915e-ff9d26cf2a6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:12.383846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383700 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526b1087-12ff-4e65-9eaa-83739c86c20b-trusted-ca\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.383846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383757 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.383846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383788 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e584897d-a9b9-424a-aba1-29a0dc184e3e-tmp\") pod \"downloads-6dd789c8b7-66qfc\" (UID: \"e584897d-a9b9-424a-aba1-29a0dc184e3e\") " pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.383846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383840 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfb4q\" (UniqueName: \"kubernetes.io/projected/e584897d-a9b9-424a-aba1-29a0dc184e3e-kube-api-access-vfb4q\") pod \"downloads-6dd789c8b7-66qfc\" (UID: \"e584897d-a9b9-424a-aba1-29a0dc184e3e\") " pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.384058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383891 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.384058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383937 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.384058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383967 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.384058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.383994 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526b1087-12ff-4e65-9eaa-83739c86c20b-image-registry-private-configuration\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.384058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384037 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-bound-sa-token\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384082 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxvc\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-kube-api-access-gbxvc\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384113 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384175 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-registry-tls\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384201 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526b1087-12ff-4e65-9eaa-83739c86c20b-installation-pull-secrets\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384238 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526b1087-12ff-4e65-9eaa-83739c86c20b-ca-trust-extracted\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384270 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.384321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.384313 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526b1087-12ff-4e65-9eaa-83739c86c20b-registry-certificates\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.386102 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.385716 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e584897d-a9b9-424a-aba1-29a0dc184e3e-tmp\") pod \"downloads-6dd789c8b7-66qfc\" (UID: \"e584897d-a9b9-424a-aba1-29a0dc184e3e\") " pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.386102 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.385782 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.386829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.386802 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.386912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.386867 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.387329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.387280 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.392107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.392084 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a7ed96f-845a-44ac-915e-ff9d26cf2a6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-jkvkf\" (UID: \"1a7ed96f-845a-44ac-915e-ff9d26cf2a6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:12.392755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.392737 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.401738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.399841 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.440058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.440026 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwk22\" (UniqueName: \"kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22\") pod \"console-7ffcbccbb7-ndg2f\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.440259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.440184 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfb4q\" (UniqueName: \"kubernetes.io/projected/e584897d-a9b9-424a-aba1-29a0dc184e3e-kube-api-access-vfb4q\") pod \"downloads-6dd789c8b7-66qfc\" (UID: \"e584897d-a9b9-424a-aba1-29a0dc184e3e\") " pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.453086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.453056 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:06:12.485592 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485542 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526b1087-12ff-4e65-9eaa-83739c86c20b-image-registry-private-configuration\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-bound-sa-token\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485618 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxvc\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-kube-api-access-gbxvc\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485658 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-registry-tls\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485683 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526b1087-12ff-4e65-9eaa-83739c86c20b-installation-pull-secrets\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485714 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526b1087-12ff-4e65-9eaa-83739c86c20b-ca-trust-extracted\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485750 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526b1087-12ff-4e65-9eaa-83739c86c20b-registry-certificates\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.485922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.485804 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526b1087-12ff-4e65-9eaa-83739c86c20b-trusted-ca\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.486511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.486479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526b1087-12ff-4e65-9eaa-83739c86c20b-ca-trust-extracted\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.486995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.486952 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526b1087-12ff-4e65-9eaa-83739c86c20b-trusted-ca\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.487128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.487103 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526b1087-12ff-4e65-9eaa-83739c86c20b-registry-certificates\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.489034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.489013 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526b1087-12ff-4e65-9eaa-83739c86c20b-installation-pull-secrets\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.489130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.489106 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526b1087-12ff-4e65-9eaa-83739c86c20b-image-registry-private-configuration\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.490679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.490659 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-registry-tls\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.495628 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.495603 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:12.506379 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.506351 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:12.512669 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.512526 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:12.539129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.539101 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxvc\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-kube-api-access-gbxvc\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.539428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.539409 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526b1087-12ff-4e65-9eaa-83739c86c20b-bound-sa-token\") pod \"image-registry-64466fbc47-7k2d9\" (UID: \"526b1087-12ff-4e65-9eaa-83739c86c20b\") " pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.599775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.599554 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:12.819527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.819494 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6m7vz-27w65"] Nov 02 08:06:12.822927 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:12.822890 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00e2e04_97be_4578_bf36_969363659bdf.slice/crio-613b58e070767214386075f1db7f91cb482fbb12b20df8e130f4a0e287cd636d WatchSource:0}: Error finding container 613b58e070767214386075f1db7f91cb482fbb12b20df8e130f4a0e287cd636d: Status 404 returned error can't find the container with id 613b58e070767214386075f1db7f91cb482fbb12b20df8e130f4a0e287cd636d Nov 02 08:06:12.923436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.923348 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-66qfc"] Nov 02 08:06:12.929288 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:12.929251 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode584897d_a9b9_424a_aba1_29a0dc184e3e.slice/crio-e6e6b32e7e77932b0463f815f2a92a760ce1711d6f1edd6894311db1ea6c124c WatchSource:0}: Error finding container e6e6b32e7e77932b0463f815f2a92a760ce1711d6f1edd6894311db1ea6c124c: Status 404 returned error can't find the container with id e6e6b32e7e77932b0463f815f2a92a760ce1711d6f1edd6894311db1ea6c124c Nov 02 08:06:12.936432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.936402 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffcbccbb7-ndg2f"] Nov 02 08:06:12.989927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.989895 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64466fbc47-7k2d9"] Nov 02 08:06:12.992689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:12.992661 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf"] Nov 02 08:06:12.996287 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:12.996249 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526b1087_12ff_4e65_9eaa_83739c86c20b.slice/crio-9ea1e192cc7ad4aff744a48dab8e4b91a321b3cb8735c459ef8bf069c66934bf WatchSource:0}: Error finding container 9ea1e192cc7ad4aff744a48dab8e4b91a321b3cb8735c459ef8bf069c66934bf: Status 404 returned error can't find the container with id 9ea1e192cc7ad4aff744a48dab8e4b91a321b3cb8735c459ef8bf069c66934bf Nov 02 08:06:12.997014 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:12.996987 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7ed96f_845a_44ac_915e_ff9d26cf2a6e.slice/crio-fba84798fb0e9b6927d7129ded67fbccc6c4a35e37ce1d1ffa659ac434ef5cc5 WatchSource:0}: Error finding container fba84798fb0e9b6927d7129ded67fbccc6c4a35e37ce1d1ffa659ac434ef5cc5: Status 404 returned error can't find the container with id fba84798fb0e9b6927d7129ded67fbccc6c4a35e37ce1d1ffa659ac434ef5cc5 Nov 02 08:06:13.296454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.296416 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trgc8" event={"ID":"ce7abe8e-c9a4-4128-a744-0a2cd073c535","Type":"ContainerStarted","Data":"d6f3ab286192492f79671ac4be97d4ce4ad42104cb668f96619113af5b28a139"} Nov 02 08:06:13.296613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.296488 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trgc8" event={"ID":"ce7abe8e-c9a4-4128-a744-0a2cd073c535","Type":"ContainerStarted","Data":"967833e7810b99bc91d410f686eb90e242d370400a928ade2711663113659dea"} Nov 02 08:06:13.301915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.301528 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6m7vz-27w65" event={"ID":"c00e2e04-97be-4578-bf36-969363659bdf","Type":"ContainerStarted","Data":"8de1f016b0f6c5a7f7ee14460e8ac4a1e2aad5772adb1037865c4b0be68d9cb4"} Nov 02 08:06:13.301915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.301580 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6m7vz-27w65" event={"ID":"c00e2e04-97be-4578-bf36-969363659bdf","Type":"ContainerStarted","Data":"613b58e070767214386075f1db7f91cb482fbb12b20df8e130f4a0e287cd636d"} Nov 02 08:06:13.302957 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.302928 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcbccbb7-ndg2f" event={"ID":"e61c5e00-5228-457e-9ad4-90e9fb7d2524","Type":"ContainerStarted","Data":"e1365c54e26cb74ece792a928522f473827fb778831a8716af1c094fef5559d7"} Nov 02 08:06:13.304086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.304061 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-66qfc" event={"ID":"e584897d-a9b9-424a-aba1-29a0dc184e3e","Type":"ContainerStarted","Data":"e6e6b32e7e77932b0463f815f2a92a760ce1711d6f1edd6894311db1ea6c124c"} Nov 02 08:06:13.305559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.305535 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" event={"ID":"1a7ed96f-845a-44ac-915e-ff9d26cf2a6e","Type":"ContainerStarted","Data":"fba84798fb0e9b6927d7129ded67fbccc6c4a35e37ce1d1ffa659ac434ef5cc5"} Nov 02 08:06:13.307613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.307588 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" event={"ID":"526b1087-12ff-4e65-9eaa-83739c86c20b","Type":"ContainerStarted","Data":"5db941eaa972d9c52a9a16e2c6fb20bd4cc2f4f95cdc6988d8adc68dc099ace1"} Nov 02 08:06:13.307713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.307630 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" event={"ID":"526b1087-12ff-4e65-9eaa-83739c86c20b","Type":"ContainerStarted","Data":"9ea1e192cc7ad4aff744a48dab8e4b91a321b3cb8735c459ef8bf069c66934bf"} Nov 02 08:06:13.307778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.307737 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:13.507210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.507128 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6m7vz-27w65" podStartSLOduration=1.507112536 podStartE2EDuration="1.507112536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:13.506594879 +0000 UTC m=+20.597215299" watchObservedRunningTime="2026-11-02 08:06:13.507112536 +0000 UTC m=+20.597732954" Nov 02 08:06:13.616226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:13.616069 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" podStartSLOduration=1.6160494330000001 podStartE2EDuration="1.616049433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:13.614606571 +0000 UTC m=+20.705226990" watchObservedRunningTime="2026-11-02 08:06:13.616049433 +0000 UTC m=+20.706670061" Nov 02 08:06:14.317251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:14.316188 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48rm2" event={"ID":"c5d3993c-5c24-4912-9113-f64e2ca954b3","Type":"ContainerStarted","Data":"bdf68e6bf8b4f632fa82677363030acfea41d9c47bd31ed2421e62d91801b47f"} Nov 02 08:06:15.047208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.047070 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q49jt_de009ad0-3b12-421f-a8bf-edfc92fd6b63/dns/0.log" Nov 02 08:06:15.244634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.244603 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q49jt_de009ad0-3b12-421f-a8bf-edfc92fd6b63/kube-rbac-proxy/0.log" Nov 02 08:06:15.325258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.325114 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" event={"ID":"1a7ed96f-845a-44ac-915e-ff9d26cf2a6e","Type":"ContainerStarted","Data":"3922580cd91206f176d3128881357884e6cd18bf3af662223d9c87d4717327dc"} Nov 02 08:06:15.325743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.325640 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:15.330871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.330843 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" Nov 02 08:06:15.430389 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.430328 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-jkvkf" podStartSLOduration=1.665055754 podStartE2EDuration="3.430308429s" totalImagesPullingTime="1.765252675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:15.428928636 +0000 UTC m=+22.519549056" watchObservedRunningTime="2026-11-02 08:06:15.430308429 +0000 UTC m=+22.520928849" Nov 02 08:06:15.643688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:15.643659 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m29jr_506e21c2-5241-44f6-8c30-0581ebe68600/dns-node-resolver/0.log" Nov 02 08:06:16.043927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:16.043871 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-595bc5b6bc-js2wl_95dd7a9d-509f-48cd-b87e-c50f688c4a9a/registry/0.log" Nov 02 08:06:16.243981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:16.243951 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59f6c46786-fzsst_2e1c4943-feda-467b-9d73-33f613a74060/registry/0.log" Nov 02 08:06:16.444060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:16.443993 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64466fbc47-7k2d9_526b1087-12ff-4e65-9eaa-83739c86c20b/registry/0.log" Nov 02 08:06:16.842958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:16.842933 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s52jj_c2ebff0e-2c89-4345-8577-9e296f299461/node-ca/0.log" Nov 02 08:06:17.243631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:17.243602 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d5df879b9-lxqd6_6045c48b-d8e9-4c92-8201-3ccc48a38a91/router/0.log" Nov 02 08:06:17.337002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:17.336956 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcbccbb7-ndg2f" event={"ID":"e61c5e00-5228-457e-9ad4-90e9fb7d2524","Type":"ContainerStarted","Data":"28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01"} Nov 02 08:06:17.444893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:17.444841 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7ffcbccbb7-ndg2f" podStartSLOduration=1.580988244 podStartE2EDuration="5.444825139s" totalImagesPullingTime="3.863836895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:17.44302581 +0000 UTC m=+24.533646229" watchObservedRunningTime="2026-11-02 08:06:17.444825139 +0000 UTC m=+24.535445557" Nov 02 08:06:17.643688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:17.643603 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-svhfk_62e8f7ea-8d81-4b80-a527-f6e228885382/serve-healthcheck-canary/0.log" Nov 02 08:06:20.065337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:20.065302 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8d2bg" Nov 02 08:06:21.750757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:21.750727 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:06:22.231709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:22.231678 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:22.513862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:22.513784 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:22.513862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:22.513839 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:22.519690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:22.519659 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:23.263406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.263375 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hlm8n" Nov 02 08:06:23.376901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.376870 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:23.475936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.475902 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nj7wb"] Nov 02 08:06:23.500172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.500126 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.564351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.564158 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbg2j\"" Nov 02 08:06:23.579636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.579601 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 08:06:23.579636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.579612 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 08:06:23.596126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596095 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-wtmp\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596132 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ffe883-92cf-4e01-a8fd-80d03b061b47-metrics-client-ca\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596267 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-root\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596307 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-accelerators-collector-config\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596379 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596413 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwldx\" (UniqueName: \"kubernetes.io/projected/d6ffe883-92cf-4e01-a8fd-80d03b061b47-kube-api-access-nwldx\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596434 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-textfile\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596460 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-sys\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.596604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596488 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-tls\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.597021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.596999 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 08:06:23.597109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.597035 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 08:06:23.698083 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698047 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-wtmp\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698087 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ffe883-92cf-4e01-a8fd-80d03b061b47-metrics-client-ca\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698126 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-root\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698174 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-accelerators-collector-config\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698241 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698284 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwldx\" (UniqueName: \"kubernetes.io/projected/d6ffe883-92cf-4e01-a8fd-80d03b061b47-kube-api-access-nwldx\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698315 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-textfile\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698341 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-sys\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.698513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.698374 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-tls\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.699583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.699557 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ffe883-92cf-4e01-a8fd-80d03b061b47-metrics-client-ca\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.699728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.699710 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-wtmp\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.699790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.699758 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-root\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.699851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.699802 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6ffe883-92cf-4e01-a8fd-80d03b061b47-sys\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.699972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.699947 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-textfile\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.700526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.700479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-accelerators-collector-config\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.701464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.701439 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-tls\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.701800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.701780 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6ffe883-92cf-4e01-a8fd-80d03b061b47-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.752075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.752039 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwldx\" (UniqueName: \"kubernetes.io/projected/d6ffe883-92cf-4e01-a8fd-80d03b061b47-kube-api-access-nwldx\") pod \"node-exporter-nj7wb\" (UID: \"d6ffe883-92cf-4e01-a8fd-80d03b061b47\") " pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.863677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:23.863594 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nj7wb" Nov 02 08:06:23.901659 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:06:23.901623 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ffe883_92cf_4e01_a8fd_80d03b061b47.slice/crio-38c9a132414d6e9354d37b140063f06533c1a520821157f8251c39b8f3c54319 WatchSource:0}: Error finding container 38c9a132414d6e9354d37b140063f06533c1a520821157f8251c39b8f3c54319: Status 404 returned error can't find the container with id 38c9a132414d6e9354d37b140063f06533c1a520821157f8251c39b8f3c54319 Nov 02 08:06:24.379443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:24.379399 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nj7wb" event={"ID":"d6ffe883-92cf-4e01-a8fd-80d03b061b47","Type":"ContainerStarted","Data":"38c9a132414d6e9354d37b140063f06533c1a520821157f8251c39b8f3c54319"} Nov 02 08:06:24.926325 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:24.926277 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:06:26.390186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:26.390077 5055 generic.go:356] "Generic (PLEG): container finished" podID="d6ffe883-92cf-4e01-a8fd-80d03b061b47" containerID="d1eea889f34bf7cd45e55649ba79fc251b235fd1c48e24f2a80aa033228dca51" exitCode=0 Nov 02 08:06:26.390632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:26.390172 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nj7wb" event={"ID":"d6ffe883-92cf-4e01-a8fd-80d03b061b47","Type":"ContainerDied","Data":"d1eea889f34bf7cd45e55649ba79fc251b235fd1c48e24f2a80aa033228dca51"} Nov 02 08:06:27.399116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:27.399076 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nj7wb" event={"ID":"d6ffe883-92cf-4e01-a8fd-80d03b061b47","Type":"ContainerStarted","Data":"c9a54b7d1794fcd6710a7d384409f805b59f8090ff626c34f885c5bee1e62428"} Nov 02 08:06:27.399649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:27.399158 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nj7wb" event={"ID":"d6ffe883-92cf-4e01-a8fd-80d03b061b47","Type":"ContainerStarted","Data":"0538bf288da1933f042f96074495c2a6c03471d5b4f7c59764fca6e30a942c9f"} Nov 02 08:06:27.598249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:27.598188 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nj7wb" podStartSLOduration=2.340306741 podStartE2EDuration="4.598175395s" totalImagesPullingTime="2.163570366s" totalInitContainerRuntime="94.298288ms" isStatefulPod=true podCreationTimestamp="2026-11-02 08:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:27.552207163 +0000 UTC m=+34.642827583" watchObservedRunningTime="2026-11-02 08:06:27.598175395 +0000 UTC m=+34.688795814" Nov 02 08:06:30.207812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:30.207772 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ffcbccbb7-ndg2f"] Nov 02 08:06:30.416790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:30.416753 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-66qfc" event={"ID":"e584897d-a9b9-424a-aba1-29a0dc184e3e","Type":"ContainerStarted","Data":"a57c9d853bf3b8b5cc73b657d7dd8ed30325213bdd060bbc78eafa984eb14682"} Nov 02 08:06:30.417026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:30.417004 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:30.431110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:30.431078 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-66qfc" Nov 02 08:06:30.536245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:30.536095 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-66qfc" podStartSLOduration=1.665823949 podStartE2EDuration="18.536075914s" totalImagesPullingTime="16.870251965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:30.535114089 +0000 UTC m=+37.625734612" watchObservedRunningTime="2026-11-02 08:06:30.536075914 +0000 UTC m=+37.626696325" Nov 02 08:06:34.323334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:34.322999 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64466fbc47-7k2d9" Nov 02 08:06:34.531539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:34.531503 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-595bc5b6bc-js2wl"] Nov 02 08:06:37.244448 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.244364 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" podUID="2e1c4943-feda-467b-9d73-33f613a74060" containerName="registry" containerID="cri-o://8a191af5c09a99f5a9b6c120efb22925be90be6870e89afb5de0c0e7d70bbb26" gracePeriod=30 Nov 02 08:06:37.461656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.461464 5055 generic.go:356] "Generic (PLEG): container finished" podID="2e1c4943-feda-467b-9d73-33f613a74060" containerID="8a191af5c09a99f5a9b6c120efb22925be90be6870e89afb5de0c0e7d70bbb26" exitCode=0 Nov 02 08:06:37.461656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.461589 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" event={"ID":"2e1c4943-feda-467b-9d73-33f613a74060","Type":"ContainerDied","Data":"8a191af5c09a99f5a9b6c120efb22925be90be6870e89afb5de0c0e7d70bbb26"} Nov 02 08:06:37.560405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.560364 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:37.639569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639532 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.639742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639586 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.639742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639680 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.639742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639721 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.639921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639753 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.639921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639791 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.639921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639822 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.640072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.639965 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:37.640526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.640501 5055 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-registry-certificates\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.642059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.642004 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:37.643132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.643102 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "kube-api-access-4d6t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:37.643252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.643230 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:37.643405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.643387 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:06:37.644556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.644532 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:37.649203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.649171 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:06:37.741075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741042 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") pod \"2e1c4943-feda-467b-9d73-33f613a74060\" (UID: \"2e1c4943-feda-467b-9d73-33f613a74060\") " Nov 02 08:06:37.741474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741284 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6t8\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-kube-api-access-4d6t8\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.741474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741303 5055 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e1c4943-feda-467b-9d73-33f613a74060-ca-trust-extracted\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.741474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741318 5055 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-registry-tls\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.741474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741329 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e1c4943-feda-467b-9d73-33f613a74060-trusted-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.741474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741338 5055 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-image-registry-private-configuration\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.741474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.741351 5055 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e1c4943-feda-467b-9d73-33f613a74060-bound-sa-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:37.743609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.743580 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets" pod "2e1c4943-feda-467b-9d73-33f613a74060" (UID: "2e1c4943-feda-467b-9d73-33f613a74060"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:06:37.842107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:37.842029 5055 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e1c4943-feda-467b-9d73-33f613a74060-installation-pull-secrets\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:38.467422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:38.467397 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" Nov 02 08:06:38.467865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:38.467388 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f6c46786-fzsst" event={"ID":"2e1c4943-feda-467b-9d73-33f613a74060","Type":"ContainerDied","Data":"c3516146e61451e04517ba0c81c3fceb06027f4cfe3d44dfd8abb35704b54a20"} Nov 02 08:06:38.467865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:38.467539 5055 scope.go:122] "RemoveContainer" containerID="8a191af5c09a99f5a9b6c120efb22925be90be6870e89afb5de0c0e7d70bbb26" Nov 02 08:06:38.666241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:38.666157 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59f6c46786-fzsst"] Nov 02 08:06:38.669718 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:38.669689 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59f6c46786-fzsst"] Nov 02 08:06:39.090023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:39.089989 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1c4943-feda-467b-9d73-33f613a74060" path="/var/lib/kubelet/pods/2e1c4943-feda-467b-9d73-33f613a74060/volumes" Nov 02 08:06:41.448440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:41.448398 5055 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hlm8n_6449c7f8-c8f4-4022-8384-a85070a6493c/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:41.867630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:41.867598 5055 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hlm8n_6449c7f8-c8f4-4022-8384-a85070a6493c/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:44.579493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:44.579453 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t29pm-98s5b"] Nov 02 08:06:44.580034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:44.579696 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-t29pm-98s5b" podUID="d86dbbbd-8799-4701-9fa5-831d1388c9d7" containerName="insights-gathering" containerID="cri-o://41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e" gracePeriod=30 Nov 02 08:06:44.923958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:44.923931 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:06:45.005691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.005655 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") pod \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " Nov 02 08:06:45.005882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.005718 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") pod \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " Nov 02 08:06:45.005882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.005834 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") pod \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " Nov 02 08:06:45.005882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.005878 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") pod \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\" (UID: \"d86dbbbd-8799-4701-9fa5-831d1388c9d7\") " Nov 02 08:06:45.006176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.006119 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path" pod "d86dbbbd-8799-4701-9fa5-831d1388c9d7" (UID: "d86dbbbd-8799-4701-9fa5-831d1388c9d7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:06:45.006176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.006130 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle" pod "d86dbbbd-8799-4701-9fa5-831d1388c9d7" (UID: "d86dbbbd-8799-4701-9fa5-831d1388c9d7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:45.006344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.006230 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle" pod "d86dbbbd-8799-4701-9fa5-831d1388c9d7" (UID: "d86dbbbd-8799-4701-9fa5-831d1388c9d7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:45.008007 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.007986 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92" pod "d86dbbbd-8799-4701-9fa5-831d1388c9d7" (UID: "d86dbbbd-8799-4701-9fa5-831d1388c9d7"). InnerVolumeSpecName "kube-api-access-ljn92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:45.106735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.106707 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn92\" (UniqueName: \"kubernetes.io/projected/d86dbbbd-8799-4701-9fa5-831d1388c9d7-kube-api-access-ljn92\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:45.106735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.106735 5055 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d86dbbbd-8799-4701-9fa5-831d1388c9d7-archives-path\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:45.106894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.106745 5055 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-service-ca-bundle\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:45.106894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.106755 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86dbbbd-8799-4701-9fa5-831d1388c9d7-trusted-ca-bundle\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:45.507962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.507923 5055 generic.go:356] "Generic (PLEG): container finished" podID="d86dbbbd-8799-4701-9fa5-831d1388c9d7" containerID="41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e" exitCode=2 Nov 02 08:06:45.508190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.507968 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t29pm-98s5b" Nov 02 08:06:45.508190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.507997 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t29pm-98s5b" event={"ID":"d86dbbbd-8799-4701-9fa5-831d1388c9d7","Type":"ContainerDied","Data":"41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e"} Nov 02 08:06:45.508190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.508046 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t29pm-98s5b" event={"ID":"d86dbbbd-8799-4701-9fa5-831d1388c9d7","Type":"ContainerDied","Data":"fdbcc81a21f74090594501b0bbd401f436460303bf184eeb8e2527146f0d56fd"} Nov 02 08:06:45.508190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.508068 5055 scope.go:122] "RemoveContainer" containerID="41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e" Nov 02 08:06:45.516915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.516892 5055 scope.go:122] "RemoveContainer" containerID="41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e" Nov 02 08:06:45.517238 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:45.517215 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e\": container with ID starting with 41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e not found: ID does not exist" containerID="41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e" Nov 02 08:06:45.517290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.517250 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e"} err="failed to get container status \"41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e\": rpc error: code = NotFound desc = could not find container \"41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e\": container with ID starting with 41b87cd866fac39469750242d6465da302f626333e07c7307774e4c60b485a6e not found: ID does not exist" Nov 02 08:06:45.660443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.660411 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t29pm-98s5b"] Nov 02 08:06:45.706498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:45.706461 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-t29pm-98s5b"] Nov 02 08:06:47.090075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:47.090041 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86dbbbd-8799-4701-9fa5-831d1388c9d7" path="/var/lib/kubelet/pods/d86dbbbd-8799-4701-9fa5-831d1388c9d7/volumes" Nov 02 08:06:54.918764 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:54.918714 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:06:55.232821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.232759 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7ffcbccbb7-ndg2f" podUID="e61c5e00-5228-457e-9ad4-90e9fb7d2524" containerName="console" containerID="cri-o://28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01" gracePeriod=15 Nov 02 08:06:55.493064 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.493004 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:55.550826 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.550792 5055 generic.go:356] "Generic (PLEG): container finished" podID="e61c5e00-5228-457e-9ad4-90e9fb7d2524" containerID="28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01" exitCode=0 Nov 02 08:06:55.550998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.550837 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcbccbb7-ndg2f" event={"ID":"e61c5e00-5228-457e-9ad4-90e9fb7d2524","Type":"ContainerDied","Data":"28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01"} Nov 02 08:06:55.550998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.550871 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcbccbb7-ndg2f" event={"ID":"e61c5e00-5228-457e-9ad4-90e9fb7d2524","Type":"ContainerDied","Data":"e1365c54e26cb74ece792a928522f473827fb778831a8716af1c094fef5559d7"} Nov 02 08:06:55.550998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.550890 5055 scope.go:122] "RemoveContainer" containerID="28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01" Nov 02 08:06:55.550998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.550837 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcbccbb7-ndg2f" Nov 02 08:06:55.558942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.558904 5055 scope.go:122] "RemoveContainer" containerID="28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01" Nov 02 08:06:55.559216 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:06:55.559197 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01\": container with ID starting with 28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01 not found: ID does not exist" containerID="28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01" Nov 02 08:06:55.559264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.559226 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01"} err="failed to get container status \"28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01\": rpc error: code = NotFound desc = could not find container \"28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01\": container with ID starting with 28efebaa025fbce94471aaee501129de72e1b0d246654720ae1eaefa25cb4d01 not found: ID does not exist" Nov 02 08:06:55.597667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597636 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.597837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597708 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22\" (UniqueName: \"kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.597837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597730 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.597837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597758 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.597837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597792 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.598051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597925 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.598051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.597970 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:06:55.598051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598008 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert\") pod \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\" (UID: \"e61c5e00-5228-457e-9ad4-90e9fb7d2524\") " Nov 02 08:06:55.598253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598226 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:55.598315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598277 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:55.598372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598344 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:55.598426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598391 5055 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.598426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598405 5055 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c5e00-5228-457e-9ad4-90e9fb7d2524-tmp\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.598426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.598414 5055 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-config\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.600087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.600056 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:06:55.600285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.600113 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:06:55.600285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.600117 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22" pod "e61c5e00-5228-457e-9ad4-90e9fb7d2524" (UID: "e61c5e00-5228-457e-9ad4-90e9fb7d2524"). InnerVolumeSpecName "kube-api-access-nwk22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:55.698818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.698784 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwk22\" (UniqueName: \"kubernetes.io/projected/e61c5e00-5228-457e-9ad4-90e9fb7d2524-kube-api-access-nwk22\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.698818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.698818 5055 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-serving-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.699010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.698833 5055 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e61c5e00-5228-457e-9ad4-90e9fb7d2524-console-oauth-config\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.699010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.698847 5055 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e61c5e00-5228-457e-9ad4-90e9fb7d2524-oauth-serving-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:55.965635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.965598 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ffcbccbb7-ndg2f"] Nov 02 08:06:55.970772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:55.970739 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ffcbccbb7-ndg2f"] Nov 02 08:06:57.089622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:57.089583 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61c5e00-5228-457e-9ad4-90e9fb7d2524" path="/var/lib/kubelet/pods/e61c5e00-5228-457e-9ad4-90e9fb7d2524/volumes" Nov 02 08:06:59.560132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.560085 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" podUID="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" containerName="registry" containerID="cri-o://4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4" gracePeriod=30 Nov 02 08:06:59.843164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.843104 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:06:59.937764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.937723 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.937955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.937842 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.937955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.937892 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.937955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.937923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.938171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938047 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.938171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938106 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.938292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938181 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.938292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938216 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") pod \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\" (UID: \"95dd7a9d-509f-48cd-b87e-c50f688c4a9a\") " Nov 02 08:06:59.938441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938401 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:59.938842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938573 5055 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-certificates\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:06:59.938842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.938790 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:06:59.940589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.940540 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "kube-api-access-n6wwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:59.940821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.940792 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:59.940981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.940960 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:06:59.941036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.940994 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:06:59.941072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.941034 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:06:59.947438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:06:59.947413 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted" pod "95dd7a9d-509f-48cd-b87e-c50f688c4a9a" (UID: "95dd7a9d-509f-48cd-b87e-c50f688c4a9a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:00.039414 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039377 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-trusted-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.039414 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039406 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wwj\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-kube-api-access-n6wwj\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.039414 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039418 5055 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-installation-pull-secrets\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.039666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039427 5055 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-bound-sa-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.039666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039436 5055 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-registry-tls\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.039666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039446 5055 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-image-registry-private-configuration\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.039666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.039455 5055 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95dd7a9d-509f-48cd-b87e-c50f688c4a9a-ca-trust-extracted\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:00.572944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.572905 5055 generic.go:356] "Generic (PLEG): container finished" podID="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" containerID="4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4" exitCode=0 Nov 02 08:07:00.573354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.572979 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" Nov 02 08:07:00.573354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.572987 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" event={"ID":"95dd7a9d-509f-48cd-b87e-c50f688c4a9a","Type":"ContainerDied","Data":"4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4"} Nov 02 08:07:00.573354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.573068 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595bc5b6bc-js2wl" event={"ID":"95dd7a9d-509f-48cd-b87e-c50f688c4a9a","Type":"ContainerDied","Data":"5791b6f6782d4af46fb0d3dc545cc567eb14fd8bfb6a56348d81c6bd8bc9f99f"} Nov 02 08:07:00.573354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.573089 5055 scope.go:122] "RemoveContainer" containerID="4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4" Nov 02 08:07:00.581851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.581830 5055 scope.go:122] "RemoveContainer" containerID="4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4" Nov 02 08:07:00.582184 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:07:00.582128 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4\": container with ID starting with 4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4 not found: ID does not exist" containerID="4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4" Nov 02 08:07:00.582287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.582191 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4"} err="failed to get container status \"4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4\": rpc error: code = NotFound desc = could not find container \"4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4\": container with ID starting with 4bbf66811295eabf3adb904508af1f3809e205f8884725e6ec357451662c17f4 not found: ID does not exist" Nov 02 08:07:00.771731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.771697 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-595bc5b6bc-js2wl"] Nov 02 08:07:00.817812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:00.817772 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-595bc5b6bc-js2wl"] Nov 02 08:07:01.089982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:01.089947 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" path="/var/lib/kubelet/pods/95dd7a9d-509f-48cd-b87e-c50f688c4a9a/volumes" Nov 02 08:07:13.628085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:13.628046 5055 generic.go:356] "Generic (PLEG): container finished" podID="05387920-8354-45bf-85e3-513462f484bf" containerID="194264e54169b934c2f347dcd469b8426cf7e450fc5a03d08b2f651dc4e09ee2" exitCode=0 Nov 02 08:07:13.628525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:13.628118 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" event={"ID":"05387920-8354-45bf-85e3-513462f484bf","Type":"ContainerDied","Data":"194264e54169b934c2f347dcd469b8426cf7e450fc5a03d08b2f651dc4e09ee2"} Nov 02 08:07:13.628525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:13.628465 5055 scope.go:122] "RemoveContainer" containerID="194264e54169b934c2f347dcd469b8426cf7e450fc5a03d08b2f651dc4e09ee2" Nov 02 08:07:14.633629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:14.633595 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hhvjx" event={"ID":"05387920-8354-45bf-85e3-513462f484bf","Type":"ContainerStarted","Data":"ce0eedfce7d64488bbcc66e625042d21c4e0609c128783ef5f6c716e47362427"} Nov 02 08:07:23.669959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:23.669926 5055 generic.go:356] "Generic (PLEG): container finished" podID="e3a3cb18-3e9a-4452-88bb-a8c0515db6fd" containerID="adc2e405949023b91e3e03cea945305eca41257b9703660bd3554bde0a761b82" exitCode=0 Nov 02 08:07:23.670499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:23.669980 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" event={"ID":"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd","Type":"ContainerDied","Data":"adc2e405949023b91e3e03cea945305eca41257b9703660bd3554bde0a761b82"} Nov 02 08:07:23.670499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:23.670383 5055 scope.go:122] "RemoveContainer" containerID="adc2e405949023b91e3e03cea945305eca41257b9703660bd3554bde0a761b82" Nov 02 08:07:24.676453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:24.676409 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w8xbd" event={"ID":"e3a3cb18-3e9a-4452-88bb-a8c0515db6fd","Type":"ContainerStarted","Data":"978e4271a3b0f27aee4b23191b69c9115e94dd26221727ab79694765dd8f2b34"} Nov 02 08:07:24.918957 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:07:24.918922 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:07:39.739724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:39.739693 5055 generic.go:356] "Generic (PLEG): container finished" podID="c00e2e04-97be-4578-bf36-969363659bdf" containerID="8de1f016b0f6c5a7f7ee14460e8ac4a1e2aad5772adb1037865c4b0be68d9cb4" exitCode=0 Nov 02 08:07:39.740194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:39.739753 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6m7vz-27w65" event={"ID":"c00e2e04-97be-4578-bf36-969363659bdf","Type":"ContainerDied","Data":"8de1f016b0f6c5a7f7ee14460e8ac4a1e2aad5772adb1037865c4b0be68d9cb4"} Nov 02 08:07:40.885916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.885891 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:07:40.980959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.980902 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path\") pod \"c00e2e04-97be-4578-bf36-969363659bdf\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " Nov 02 08:07:40.980959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.980963 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt\") pod \"c00e2e04-97be-4578-bf36-969363659bdf\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " Nov 02 08:07:40.981295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.980988 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle\") pod \"c00e2e04-97be-4578-bf36-969363659bdf\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " Nov 02 08:07:40.981295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.981027 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle\") pod \"c00e2e04-97be-4578-bf36-969363659bdf\" (UID: \"c00e2e04-97be-4578-bf36-969363659bdf\") " Nov 02 08:07:40.981501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.981457 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path" pod "c00e2e04-97be-4578-bf36-969363659bdf" (UID: "c00e2e04-97be-4578-bf36-969363659bdf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:40.981543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.981468 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle" pod "c00e2e04-97be-4578-bf36-969363659bdf" (UID: "c00e2e04-97be-4578-bf36-969363659bdf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:40.981543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.981485 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle" pod "c00e2e04-97be-4578-bf36-969363659bdf" (UID: "c00e2e04-97be-4578-bf36-969363659bdf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:40.983652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:40.983619 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt" pod "c00e2e04-97be-4578-bf36-969363659bdf" (UID: "c00e2e04-97be-4578-bf36-969363659bdf"). InnerVolumeSpecName "kube-api-access-h2stt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:41.081862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.081740 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-trusted-ca-bundle\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:41.081862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.081785 5055 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c00e2e04-97be-4578-bf36-969363659bdf-archives-path\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:41.081862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.081799 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/c00e2e04-97be-4578-bf36-969363659bdf-kube-api-access-h2stt\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:41.081862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.081813 5055 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00e2e04-97be-4578-bf36-969363659bdf-service-ca-bundle\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:41.748785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.748756 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6m7vz-27w65" Nov 02 08:07:41.748965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.748758 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6m7vz-27w65" event={"ID":"c00e2e04-97be-4578-bf36-969363659bdf","Type":"ContainerDied","Data":"613b58e070767214386075f1db7f91cb482fbb12b20df8e130f4a0e287cd636d"} Nov 02 08:07:41.748965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:07:41.748903 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613b58e070767214386075f1db7f91cb482fbb12b20df8e130f4a0e287cd636d" Nov 02 08:07:54.918219 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:07:54.918180 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:08:24.919985 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:08:24.919896 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:08:54.918736 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:08:54.918699 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:09:24.918627 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:09:24.918583 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:09:47.666657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666621 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl"] Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666932 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d86dbbbd-8799-4701-9fa5-831d1388c9d7" containerName="insights-gathering" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666942 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86dbbbd-8799-4701-9fa5-831d1388c9d7" containerName="insights-gathering" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666957 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e61c5e00-5228-457e-9ad4-90e9fb7d2524" containerName="console" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666962 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61c5e00-5228-457e-9ad4-90e9fb7d2524" containerName="console" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666975 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2e1c4943-feda-467b-9d73-33f613a74060" containerName="registry" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666980 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1c4943-feda-467b-9d73-33f613a74060" containerName="registry" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666986 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c00e2e04-97be-4578-bf36-969363659bdf" containerName="insights-gathering" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.666991 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00e2e04-97be-4578-bf36-969363659bdf" containerName="insights-gathering" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667000 5055 cpu_manager.go:414] "containerMap: removing container" podUID="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" containerName="registry" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667004 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" containerName="registry" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667072 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86dbbbd-8799-4701-9fa5-831d1388c9d7" containerName="insights-gathering" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667080 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61c5e00-5228-457e-9ad4-90e9fb7d2524" containerName="console" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667087 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1c4943-feda-467b-9d73-33f613a74060" containerName="registry" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667092 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00e2e04-97be-4578-bf36-969363659bdf" containerName="insights-gathering" Nov 02 08:09:47.667097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.667097 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="95dd7a9d-509f-48cd-b87e-c50f688c4a9a" containerName="registry" Nov 02 08:09:47.669247 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.669229 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.717784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.717747 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 08:09:47.744221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.744189 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zmgsp\"" Nov 02 08:09:47.769607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.769570 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl"] Nov 02 08:09:47.793273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793229 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de352b-640f-49b5-873b-1f8873d031b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793354 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793393 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793427 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de352b-640f-49b5-873b-1f8873d031b1-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793484 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793506 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de352b-640f-49b5-873b-1f8873d031b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lx9\" (UniqueName: \"kubernetes.io/projected/d7de352b-640f-49b5-873b-1f8873d031b1-kube-api-access-s4lx9\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793669 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.793720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.793701 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894559 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894600 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894623 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de352b-640f-49b5-873b-1f8873d031b1-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894660 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894680 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de352b-640f-49b5-873b-1f8873d031b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894736 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lx9\" (UniqueName: \"kubernetes.io/projected/d7de352b-640f-49b5-873b-1f8873d031b1-kube-api-access-s4lx9\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894760 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894788 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.894854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894827 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de352b-640f-49b5-873b-1f8873d031b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.895257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.894994 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.895257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.895065 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.895359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.895308 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.895416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.895395 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.895541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.895521 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de352b-640f-49b5-873b-1f8873d031b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.897122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.897091 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de352b-640f-49b5-873b-1f8873d031b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.897250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.897233 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de352b-640f-49b5-873b-1f8873d031b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.948388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.948314 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de352b-640f-49b5-873b-1f8873d031b1-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.948513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.948432 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lx9\" (UniqueName: \"kubernetes.io/projected/d7de352b-640f-49b5-873b-1f8873d031b1-kube-api-access-s4lx9\") pod \"maas-default-gateway-openshift-default-769df7846c-tskjl\" (UID: \"d7de352b-640f-49b5-873b-1f8873d031b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.980431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.980396 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:47.983555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.983020 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f"] Nov 02 08:09:47.985561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:47.985538 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.033872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.033720 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-mgj2c\"" Nov 02 08:09:48.048018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.047931 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f"] Nov 02 08:09:48.097008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.096971 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btk6g\" (UniqueName: \"kubernetes.io/projected/b662ed93-4c1c-4bd4-986f-68248f334b43-kube-api-access-btk6g\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097009 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662ed93-4c1c-4bd4-986f-68248f334b43-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097038 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097176 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097208 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097224 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097324 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097364 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.097433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.097408 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198095 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198184 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198209 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198224 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198261 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198287 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198349 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198389 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk6g\" (UniqueName: \"kubernetes.io/projected/b662ed93-4c1c-4bd4-986f-68248f334b43-kube-api-access-btk6g\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198425 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662ed93-4c1c-4bd4-986f-68248f334b43-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198620 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198639 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.198732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198716 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.199037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.198882 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.199104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.199049 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662ed93-4c1c-4bd4-986f-68248f334b43-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.200467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.200440 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.201047 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.201031 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.251462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.251431 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662ed93-4c1c-4bd4-986f-68248f334b43-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.251639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.251620 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk6g\" (UniqueName: \"kubernetes.io/projected/b662ed93-4c1c-4bd4-986f-68248f334b43-kube-api-access-btk6g\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f\" (UID: \"b662ed93-4c1c-4bd4-986f-68248f334b43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.306839 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.306793 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:48.401437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.401396 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl"] Nov 02 08:09:48.405235 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:09:48.405206 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7de352b_640f_49b5_873b_1f8873d031b1.slice/crio-57afb56d60f6f8ee94b4c14f57390c8d673a41f99c6ee798911294e6802a2061 WatchSource:0}: Error finding container 57afb56d60f6f8ee94b4c14f57390c8d673a41f99c6ee798911294e6802a2061: Status 404 returned error can't find the container with id 57afb56d60f6f8ee94b4c14f57390c8d673a41f99c6ee798911294e6802a2061 Nov 02 08:09:48.671663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:48.671589 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f"] Nov 02 08:09:48.674422 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:09:48.674387 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb662ed93_4c1c_4bd4_986f_68248f334b43.slice/crio-0e9c5c8042df86d34e3f83accf3abec03bbf7f1d50f08a0a9a7789e417c14223 WatchSource:0}: Error finding container 0e9c5c8042df86d34e3f83accf3abec03bbf7f1d50f08a0a9a7789e417c14223: Status 404 returned error can't find the container with id 0e9c5c8042df86d34e3f83accf3abec03bbf7f1d50f08a0a9a7789e417c14223 Nov 02 08:09:49.250220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:49.250170 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" event={"ID":"b662ed93-4c1c-4bd4-986f-68248f334b43","Type":"ContainerStarted","Data":"0e9c5c8042df86d34e3f83accf3abec03bbf7f1d50f08a0a9a7789e417c14223"} Nov 02 08:09:49.251267 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:49.251242 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" event={"ID":"d7de352b-640f-49b5-873b-1f8873d031b1","Type":"ContainerStarted","Data":"57afb56d60f6f8ee94b4c14f57390c8d673a41f99c6ee798911294e6802a2061"} Nov 02 08:09:52.988790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:52.988157 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:09:52.988790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:52.988246 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:09:52.988790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:52.988293 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:09:53.239313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:53.239272 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:09:53.239402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:53.239375 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:09:53.239453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:53.239421 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:09:53.274847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:53.274807 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" event={"ID":"d7de352b-640f-49b5-873b-1f8873d031b1","Type":"ContainerStarted","Data":"70db185696ddaba8f5d5bec00676ed4fc818b4fb194cd70d346420608cb8f32b"} Nov 02 08:09:53.398501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:53.398446 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" podStartSLOduration=1.81813007 podStartE2EDuration="6.398427255s" totalImagesPullingTime="4.580297185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:53.394495803 +0000 UTC m=+240.485116221" watchObservedRunningTime="2026-11-02 08:09:53.398427255 +0000 UTC m=+240.489047671" Nov 02 08:09:53.981391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:53.981353 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:54.279955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.279865 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" event={"ID":"b662ed93-4c1c-4bd4-986f-68248f334b43","Type":"ContainerStarted","Data":"116f76853b5ffe58add7b957404440f5928988329330fdd3109b997a11acc7c3"} Nov 02 08:09:54.307472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.307435 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:54.311952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.311925 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:54.435644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.435582 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" podStartSLOduration=2.873606682 podStartE2EDuration="7.435565233s" totalImagesPullingTime="4.561958551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:54.434732786 +0000 UTC m=+241.525353215" watchObservedRunningTime="2026-11-02 08:09:54.435565233 +0000 UTC m=+241.526185652" Nov 02 08:09:54.481951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.481916 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x"] Nov 02 08:09:54.484990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.484967 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.533657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.533575 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 08:09:54.533832 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.533662 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 08:09:54.533832 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.533662 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-m98s7\"" Nov 02 08:09:54.542376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.542345 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x"] Nov 02 08:09:54.660062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.660031 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.660062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.660066 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtvl\" (UniqueName: \"kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.660335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.660090 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.760882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.760838 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.760882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.760888 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtvl\" (UniqueName: \"kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.761116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.761012 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.761365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.761347 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.761400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.761371 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.814095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.814032 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtvl\" (UniqueName: \"kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:54.918323 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:09:54.918280 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:09:54.985475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:54.985443 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:55.098970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:55.098881 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:09:55.284468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:55.284435 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:55.284908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:55.284496 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:55.285316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:55.285296 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:09:55.285471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:55.285453 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f" Nov 02 08:09:55.539825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:55.539626 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x"] Nov 02 08:09:55.543524 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:09:55.543378 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad1688e_0b04_46ff_9a4b_19b97533a4af.slice/crio-6c8fafb4f12d215c3e109447136ed8d9bdcef7614591cbf8542058da8c449c66 WatchSource:0}: Error finding container 6c8fafb4f12d215c3e109447136ed8d9bdcef7614591cbf8542058da8c449c66: Status 404 returned error can't find the container with id 6c8fafb4f12d215c3e109447136ed8d9bdcef7614591cbf8542058da8c449c66 Nov 02 08:09:56.290004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:09:56.289960 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" event={"ID":"0ad1688e-0b04-46ff-9a4b-19b97533a4af","Type":"ContainerStarted","Data":"6c8fafb4f12d215c3e109447136ed8d9bdcef7614591cbf8542058da8c449c66"} Nov 02 08:10:02.319864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:02.319825 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerID="9036025223ea402672271a33399e3899bc99bef421693e9badb17c2a717ed751" exitCode=0 Nov 02 08:10:02.320317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:02.319882 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" event={"ID":"0ad1688e-0b04-46ff-9a4b-19b97533a4af","Type":"ContainerDied","Data":"9036025223ea402672271a33399e3899bc99bef421693e9badb17c2a717ed751"} Nov 02 08:10:06.340189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:06.340129 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerID="fd147f58f6f1ab30d95656d079a350991e31354fe3dfe09cedabe146cb338774" exitCode=0 Nov 02 08:10:06.340189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:06.340176 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" event={"ID":"0ad1688e-0b04-46ff-9a4b-19b97533a4af","Type":"ContainerDied","Data":"fd147f58f6f1ab30d95656d079a350991e31354fe3dfe09cedabe146cb338774"} Nov 02 08:10:13.373329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:13.373244 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" event={"ID":"0ad1688e-0b04-46ff-9a4b-19b97533a4af","Type":"ContainerStarted","Data":"df7f3c606ed684adfaa15a667458dacab938bb5dd8bd6a3c6f0f0a2146fe9f53"} Nov 02 08:10:13.481008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:13.480952 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" podStartSLOduration=3.418556467 podStartE2EDuration="19.480936582s" totalImagesPullingTime="16.062380115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:13.479880106 +0000 UTC m=+260.570500526" watchObservedRunningTime="2026-11-02 08:10:13.480936582 +0000 UTC m=+260.571557003" Nov 02 08:10:14.379694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:14.379653 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerID="df7f3c606ed684adfaa15a667458dacab938bb5dd8bd6a3c6f0f0a2146fe9f53" exitCode=0 Nov 02 08:10:14.380091 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:14.379694 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" event={"ID":"0ad1688e-0b04-46ff-9a4b-19b97533a4af","Type":"ContainerDied","Data":"df7f3c606ed684adfaa15a667458dacab938bb5dd8bd6a3c6f0f0a2146fe9f53"} Nov 02 08:10:15.573853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.573821 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:10:15.623356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.623324 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util\") pod \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " Nov 02 08:10:15.623514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.623359 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle\") pod \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " Nov 02 08:10:15.623514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.623391 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl\" (UniqueName: \"kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl\") pod \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\" (UID: \"0ad1688e-0b04-46ff-9a4b-19b97533a4af\") " Nov 02 08:10:15.623990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.623957 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle" pod "0ad1688e-0b04-46ff-9a4b-19b97533a4af" (UID: "0ad1688e-0b04-46ff-9a4b-19b97533a4af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:15.625734 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.625696 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl" pod "0ad1688e-0b04-46ff-9a4b-19b97533a4af" (UID: "0ad1688e-0b04-46ff-9a4b-19b97533a4af"). InnerVolumeSpecName "kube-api-access-4jtvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:15.628967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.628938 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util" pod "0ad1688e-0b04-46ff-9a4b-19b97533a4af" (UID: "0ad1688e-0b04-46ff-9a4b-19b97533a4af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:15.723909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.723871 5055 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-util\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:15.723909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.723902 5055 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ad1688e-0b04-46ff-9a4b-19b97533a4af-bundle\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:15.723909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:15.723914 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtvl\" (UniqueName: \"kubernetes.io/projected/0ad1688e-0b04-46ff-9a4b-19b97533a4af-kube-api-access-4jtvl\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:16.390437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:16.390401 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" event={"ID":"0ad1688e-0b04-46ff-9a4b-19b97533a4af","Type":"ContainerDied","Data":"6c8fafb4f12d215c3e109447136ed8d9bdcef7614591cbf8542058da8c449c66"} Nov 02 08:10:16.390616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:16.390452 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8fafb4f12d215c3e109447136ed8d9bdcef7614591cbf8542058da8c449c66" Nov 02 08:10:16.390616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:16.390450 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpb86x" Nov 02 08:10:22.637059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637026 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd"] Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637375 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="pull" Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637387 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="pull" Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637404 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="util" Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637410 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="util" Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637425 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="extract" Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637431 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="extract" Nov 02 08:10:22.637500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.637488 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad1688e-0b04-46ff-9a4b-19b97533a4af" containerName="extract" Nov 02 08:10:22.639654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.639638 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.683402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.683366 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9fca81a-c62f-4ab0-a224-f6b07323aa62-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd\" (UID: \"f9fca81a-c62f-4ab0-a224-f6b07323aa62\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.683568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.683432 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgtq\" (UniqueName: \"kubernetes.io/projected/f9fca81a-c62f-4ab0-a224-f6b07323aa62-kube-api-access-6tgtq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd\" (UID: \"f9fca81a-c62f-4ab0-a224-f6b07323aa62\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.689712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.689681 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd"] Nov 02 08:10:22.734168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.734111 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:10:22.734168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.734120 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 08:10:22.734168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.734167 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zs2wx\"" Nov 02 08:10:22.784612 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.784580 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9fca81a-c62f-4ab0-a224-f6b07323aa62-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd\" (UID: \"f9fca81a-c62f-4ab0-a224-f6b07323aa62\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.784789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.784636 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgtq\" (UniqueName: \"kubernetes.io/projected/f9fca81a-c62f-4ab0-a224-f6b07323aa62-kube-api-access-6tgtq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd\" (UID: \"f9fca81a-c62f-4ab0-a224-f6b07323aa62\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.785000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.784978 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9fca81a-c62f-4ab0-a224-f6b07323aa62-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd\" (UID: \"f9fca81a-c62f-4ab0-a224-f6b07323aa62\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.840566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.840525 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgtq\" (UniqueName: \"kubernetes.io/projected/f9fca81a-c62f-4ab0-a224-f6b07323aa62-kube-api-access-6tgtq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd\" (UID: \"f9fca81a-c62f-4ab0-a224-f6b07323aa62\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:22.950315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:22.950286 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" Nov 02 08:10:23.412230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:23.412153 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd"] Nov 02 08:10:23.415646 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:10:23.415613 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fca81a_c62f_4ab0_a224_f6b07323aa62.slice/crio-2f18f635d4d2508a590d5100f7483c74cbaa2234690de40e66f9778748596085 WatchSource:0}: Error finding container 2f18f635d4d2508a590d5100f7483c74cbaa2234690de40e66f9778748596085: Status 404 returned error can't find the container with id 2f18f635d4d2508a590d5100f7483c74cbaa2234690de40e66f9778748596085 Nov 02 08:10:23.422316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:23.422285 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" event={"ID":"f9fca81a-c62f-4ab0-a224-f6b07323aa62","Type":"ContainerStarted","Data":"2f18f635d4d2508a590d5100f7483c74cbaa2234690de40e66f9778748596085"} Nov 02 08:10:24.919961 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:10:24.919921 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:10:26.436942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:26.436907 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" event={"ID":"f9fca81a-c62f-4ab0-a224-f6b07323aa62","Type":"ContainerStarted","Data":"80b1477f0372b396b0db768fd5248ac43e952f8408f82878991d69998b1cf49d"} Nov 02 08:10:26.637764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:26.637661 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dxrhd" podStartSLOduration=1.82387974 podStartE2EDuration="4.637646132s" totalImagesPullingTime="2.813766392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:26.636560038 +0000 UTC m=+273.727180458" watchObservedRunningTime="2026-11-02 08:10:26.637646132 +0000 UTC m=+273.728266550" Nov 02 08:10:34.049619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.049575 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj"] Nov 02 08:10:34.052403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.052384 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.153361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.153323 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 08:10:34.153526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.153412 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-2kfnz\"" Nov 02 08:10:34.153526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.153476 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 08:10:34.154894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.154865 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj"] Nov 02 08:10:34.185371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.185331 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c7g\" (UniqueName: \"kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.185559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.185384 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.185559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.185474 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.286105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.286065 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c7g\" (UniqueName: \"kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.286327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.286117 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.286327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.286197 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.286651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.286625 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.286748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.286632 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.342646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.342577 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c7g\" (UniqueName: \"kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.365274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.365236 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:34.747559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.747520 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7"] Nov 02 08:10:34.750315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.750291 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj"] Nov 02 08:10:34.750467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.750401 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.774461 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:10:34.774423 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052897bf_532b_4d31_b49c_e6319dc560be.slice/crio-d4e689a54c691283b7b46d39ba227df76d496050605f655c014e362069f17173 WatchSource:0}: Error finding container d4e689a54c691283b7b46d39ba227df76d496050605f655c014e362069f17173: Status 404 returned error can't find the container with id d4e689a54c691283b7b46d39ba227df76d496050605f655c014e362069f17173 Nov 02 08:10:34.791123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.791098 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-metrics-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.791239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.791134 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.791239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.791174 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0444d31b-6109-46c2-94ce-052305a30c0c-manifests\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.791320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.791238 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npj6\" (UniqueName: \"kubernetes.io/projected/0444d31b-6109-46c2-94ce-052305a30c0c-kube-api-access-8npj6\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.791354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.791332 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-webhook-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.791402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.791389 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0444d31b-6109-46c2-94ce-052305a30c0c-tmp\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.800831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.800806 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 08:10:34.800938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.800896 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-x42xc\"" Nov 02 08:10:34.801313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.801296 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 08:10:34.801372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.801308 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 08:10:34.803248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.803224 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 08:10:34.803331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.803274 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 08:10:34.851575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.851537 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7"] Nov 02 08:10:34.892162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892111 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0444d31b-6109-46c2-94ce-052305a30c0c-tmp\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.892313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892221 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-metrics-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.892313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892252 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.892313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0444d31b-6109-46c2-94ce-052305a30c0c-manifests\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.892313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892302 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npj6\" (UniqueName: \"kubernetes.io/projected/0444d31b-6109-46c2-94ce-052305a30c0c-kube-api-access-8npj6\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.892524 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892335 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-webhook-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.892713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.892676 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0444d31b-6109-46c2-94ce-052305a30c0c-manifests\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.894372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.894337 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0444d31b-6109-46c2-94ce-052305a30c0c-tmp\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.894721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.894699 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-metrics-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.894852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.894826 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.894912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.894841 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0444d31b-6109-46c2-94ce-052305a30c0c-webhook-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:34.946118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:34.946080 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npj6\" (UniqueName: \"kubernetes.io/projected/0444d31b-6109-46c2-94ce-052305a30c0c-kube-api-access-8npj6\") pod \"opendatahub-operator-controller-manager-b679c64f6-m25l7\" (UID: \"0444d31b-6109-46c2-94ce-052305a30c0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:35.063449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:35.063360 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:35.475225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:35.475184 5055 generic.go:356] "Generic (PLEG): container finished" podID="052897bf-532b-4d31-b49c-e6319dc560be" containerID="f059b346f59c0284ee6b1be344aedbd79d1c727a3f39212347e551aef3ce6338" exitCode=0 Nov 02 08:10:35.475397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:35.475254 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" event={"ID":"052897bf-532b-4d31-b49c-e6319dc560be","Type":"ContainerDied","Data":"f059b346f59c0284ee6b1be344aedbd79d1c727a3f39212347e551aef3ce6338"} Nov 02 08:10:35.475397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:35.475297 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" event={"ID":"052897bf-532b-4d31-b49c-e6319dc560be","Type":"ContainerStarted","Data":"d4e689a54c691283b7b46d39ba227df76d496050605f655c014e362069f17173"} Nov 02 08:10:35.541881 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:10:35.541802 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0444d31b_6109_46c2_94ce_052305a30c0c.slice/crio-d33082f33305b32d8c1170b81ff15955aea18714959dd635d8312db15966d54f WatchSource:0}: Error finding container d33082f33305b32d8c1170b81ff15955aea18714959dd635d8312db15966d54f: Status 404 returned error can't find the container with id d33082f33305b32d8c1170b81ff15955aea18714959dd635d8312db15966d54f Nov 02 08:10:35.572069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:35.572028 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7"] Nov 02 08:10:36.480790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:36.480748 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" event={"ID":"0444d31b-6109-46c2-94ce-052305a30c0c","Type":"ContainerStarted","Data":"d33082f33305b32d8c1170b81ff15955aea18714959dd635d8312db15966d54f"} Nov 02 08:10:39.499602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:39.499566 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" event={"ID":"0444d31b-6109-46c2-94ce-052305a30c0c","Type":"ContainerStarted","Data":"9a267ffa6b8fc488385218c75d08fd14aecf1c0150df53bbeca3b86433d0d07d"} Nov 02 08:10:39.501355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:39.501326 5055 generic.go:356] "Generic (PLEG): container finished" podID="052897bf-532b-4d31-b49c-e6319dc560be" containerID="451cf61230560d1c0b7ddf54882684c3608ea2dea8b1e12396ae93c6226fb9e6" exitCode=0 Nov 02 08:10:39.501494 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:39.501379 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" event={"ID":"052897bf-532b-4d31-b49c-e6319dc560be","Type":"ContainerDied","Data":"451cf61230560d1c0b7ddf54882684c3608ea2dea8b1e12396ae93c6226fb9e6"} Nov 02 08:10:40.513009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:40.512969 5055 generic.go:356] "Generic (PLEG): container finished" podID="052897bf-532b-4d31-b49c-e6319dc560be" containerID="8f267891d4ae2269f181368bfe08b158a783d21121f1419c74eea8f1fc97a3de" exitCode=0 Nov 02 08:10:40.513456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:40.513047 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" event={"ID":"052897bf-532b-4d31-b49c-e6319dc560be","Type":"ContainerDied","Data":"8f267891d4ae2269f181368bfe08b158a783d21121f1419c74eea8f1fc97a3de"} Nov 02 08:10:40.514840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:40.514817 5055 generic.go:356] "Generic (PLEG): container finished" podID="0444d31b-6109-46c2-94ce-052305a30c0c" containerID="9a267ffa6b8fc488385218c75d08fd14aecf1c0150df53bbeca3b86433d0d07d" exitCode=0 Nov 02 08:10:40.514896 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:40.514850 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" event={"ID":"0444d31b-6109-46c2-94ce-052305a30c0c","Type":"ContainerDied","Data":"9a267ffa6b8fc488385218c75d08fd14aecf1c0150df53bbeca3b86433d0d07d"} Nov 02 08:10:41.668282 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.668247 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:41.756209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.756075 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle\") pod \"052897bf-532b-4d31-b49c-e6319dc560be\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " Nov 02 08:10:41.756209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.756171 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g\" (UniqueName: \"kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g\") pod \"052897bf-532b-4d31-b49c-e6319dc560be\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " Nov 02 08:10:41.756209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.756204 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util\") pod \"052897bf-532b-4d31-b49c-e6319dc560be\" (UID: \"052897bf-532b-4d31-b49c-e6319dc560be\") " Nov 02 08:10:41.756586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.756556 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle" pod "052897bf-532b-4d31-b49c-e6319dc560be" (UID: "052897bf-532b-4d31-b49c-e6319dc560be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:41.758508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.758479 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g" pod "052897bf-532b-4d31-b49c-e6319dc560be" (UID: "052897bf-532b-4d31-b49c-e6319dc560be"). InnerVolumeSpecName "kube-api-access-86c7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:41.760535 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.760512 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util" pod "052897bf-532b-4d31-b49c-e6319dc560be" (UID: "052897bf-532b-4d31-b49c-e6319dc560be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:41.856989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.856945 5055 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-bundle\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:41.856989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.856982 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86c7g\" (UniqueName: \"kubernetes.io/projected/052897bf-532b-4d31-b49c-e6319dc560be-kube-api-access-86c7g\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:41.856989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:41.856992 5055 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052897bf-532b-4d31-b49c-e6319dc560be-util\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:42.530978 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:42.530940 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" event={"ID":"0444d31b-6109-46c2-94ce-052305a30c0c","Type":"ContainerStarted","Data":"7f63fdd5bd9a3d63f8482dc11823ea9d41cc2209e4ed132536e844f12fa3c2d2"} Nov 02 08:10:42.531196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:42.531059 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:42.532919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:42.532899 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" Nov 02 08:10:42.533059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:42.532898 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfpsj" event={"ID":"052897bf-532b-4d31-b49c-e6319dc560be","Type":"ContainerDied","Data":"d4e689a54c691283b7b46d39ba227df76d496050605f655c014e362069f17173"} Nov 02 08:10:42.533059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:42.533016 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e689a54c691283b7b46d39ba227df76d496050605f655c014e362069f17173" Nov 02 08:10:42.796239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:42.796109 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" podStartSLOduration=3.582396791 podStartE2EDuration="8.796094734s" totalImagesPullingTime="4.800065519s" totalInitContainerRuntime="413.632424ms" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:42.794953933 +0000 UTC m=+289.885574352" watchObservedRunningTime="2026-11-02 08:10:42.796094734 +0000 UTC m=+289.886715147" Nov 02 08:10:44.381628 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.381578 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s"] Nov 02 08:10:44.382368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382344 5055 cpu_manager.go:414] "containerMap: removing container" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="util" Nov 02 08:10:44.382479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382369 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="util" Nov 02 08:10:44.382479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382393 5055 cpu_manager.go:414] "containerMap: removing container" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="pull" Nov 02 08:10:44.382479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382401 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="pull" Nov 02 08:10:44.382479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382417 5055 cpu_manager.go:414] "containerMap: removing container" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="extract" Nov 02 08:10:44.382479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382425 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="extract" Nov 02 08:10:44.382689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.382609 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="052897bf-532b-4d31-b49c-e6319dc560be" containerName="extract" Nov 02 08:10:44.386904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.386876 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:10:44.442079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.442045 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v42gk\"" Nov 02 08:10:44.486127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.486073 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpsz\" (UniqueName: \"kubernetes.io/projected/ca48b880-48ee-4744-8875-786ee567e6e5-kube-api-access-5rpsz\") pod \"dns-operator-controller-manager-8557cbdb84-hqt9s\" (UID: \"ca48b880-48ee-4744-8875-786ee567e6e5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:10:44.488365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.488338 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 08:10:44.488513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.488402 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 08:10:44.488513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.488435 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 08:10:44.490092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.490070 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s"] Nov 02 08:10:44.587748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.587702 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpsz\" (UniqueName: \"kubernetes.io/projected/ca48b880-48ee-4744-8875-786ee567e6e5-kube-api-access-5rpsz\") pod \"dns-operator-controller-manager-8557cbdb84-hqt9s\" (UID: \"ca48b880-48ee-4744-8875-786ee567e6e5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:10:44.647757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.647687 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpsz\" (UniqueName: \"kubernetes.io/projected/ca48b880-48ee-4744-8875-786ee567e6e5-kube-api-access-5rpsz\") pod \"dns-operator-controller-manager-8557cbdb84-hqt9s\" (UID: \"ca48b880-48ee-4744-8875-786ee567e6e5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:10:44.700210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:44.700171 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:10:45.070425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:45.070400 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s"] Nov 02 08:10:45.073294 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:10:45.073257 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca48b880_48ee_4744_8875_786ee567e6e5.slice/crio-3d12b6ad3d96a5d84cbdda83e27c08584b8f8e79c90893a049ee5c17d6285cf7 WatchSource:0}: Error finding container 3d12b6ad3d96a5d84cbdda83e27c08584b8f8e79c90893a049ee5c17d6285cf7: Status 404 returned error can't find the container with id 3d12b6ad3d96a5d84cbdda83e27c08584b8f8e79c90893a049ee5c17d6285cf7 Nov 02 08:10:45.547736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:45.547700 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" event={"ID":"ca48b880-48ee-4744-8875-786ee567e6e5","Type":"ContainerStarted","Data":"3d12b6ad3d96a5d84cbdda83e27c08584b8f8e79c90893a049ee5c17d6285cf7"} Nov 02 08:10:48.948622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:48.948594 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h"] Nov 02 08:10:48.951107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:48.951087 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:10:48.999444 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:48.999399 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wqn5s\"" Nov 02 08:10:49.020042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.020007 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h"] Nov 02 08:10:49.029596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.029559 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng8m\" (UniqueName: \"kubernetes.io/projected/e8a57647-59cb-49ba-a734-82752c851371-kube-api-access-xng8m\") pod \"authorino-operator-7b7bb4bc77-x7d9h\" (UID: \"e8a57647-59cb-49ba-a734-82752c851371\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:10:49.131051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.130970 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng8m\" (UniqueName: \"kubernetes.io/projected/e8a57647-59cb-49ba-a734-82752c851371-kube-api-access-xng8m\") pod \"authorino-operator-7b7bb4bc77-x7d9h\" (UID: \"e8a57647-59cb-49ba-a734-82752c851371\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:10:49.187758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.187723 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng8m\" (UniqueName: \"kubernetes.io/projected/e8a57647-59cb-49ba-a734-82752c851371-kube-api-access-xng8m\") pod \"authorino-operator-7b7bb4bc77-x7d9h\" (UID: \"e8a57647-59cb-49ba-a734-82752c851371\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:10:49.262500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.262458 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:10:49.567645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.567610 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" event={"ID":"ca48b880-48ee-4744-8875-786ee567e6e5","Type":"ContainerStarted","Data":"5ea4c2d3598c72066c822a5931803c493471e5a06ef2f411063b24dafaee38d1"} Nov 02 08:10:49.567818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.567685 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:10:49.627024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.626944 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h"] Nov 02 08:10:49.629821 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:10:49.629795 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a57647_59cb_49ba_a734_82752c851371.slice/crio-80ff1da489def6cad2cd612d34c26308013fc0b1d1e6abd1f3ccb18bbce267c5 WatchSource:0}: Error finding container 80ff1da489def6cad2cd612d34c26308013fc0b1d1e6abd1f3ccb18bbce267c5: Status 404 returned error can't find the container with id 80ff1da489def6cad2cd612d34c26308013fc0b1d1e6abd1f3ccb18bbce267c5 Nov 02 08:10:49.680086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:49.680033 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" podStartSLOduration=1.889469739 podStartE2EDuration="5.680019485s" totalImagesPullingTime="3.790549746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:49.677950274 +0000 UTC m=+296.768570692" watchObservedRunningTime="2026-11-02 08:10:49.680019485 +0000 UTC m=+296.770639903" Nov 02 08:10:50.573517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:50.573481 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" event={"ID":"e8a57647-59cb-49ba-a734-82752c851371","Type":"ContainerStarted","Data":"80ff1da489def6cad2cd612d34c26308013fc0b1d1e6abd1f3ccb18bbce267c5"} Nov 02 08:10:52.585345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.585315 5055 generic.go:356] "Generic (PLEG): container finished" podID="0444d31b-6109-46c2-94ce-052305a30c0c" containerID="7f63fdd5bd9a3d63f8482dc11823ea9d41cc2209e4ed132536e844f12fa3c2d2" exitCode=1 Nov 02 08:10:52.585682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.585392 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" event={"ID":"0444d31b-6109-46c2-94ce-052305a30c0c","Type":"ContainerDied","Data":"7f63fdd5bd9a3d63f8482dc11823ea9d41cc2209e4ed132536e844f12fa3c2d2"} Nov 02 08:10:52.585820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.585803 5055 scope.go:122] "RemoveContainer" containerID="7f63fdd5bd9a3d63f8482dc11823ea9d41cc2209e4ed132536e844f12fa3c2d2" Nov 02 08:10:52.866097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.866012 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb"] Nov 02 08:10:52.868723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.868706 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:10:52.917989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.917958 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kvghs\"" Nov 02 08:10:52.923619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.923586 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb"] Nov 02 08:10:52.970177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:52.970126 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpb9\" (UniqueName: \"kubernetes.io/projected/1a2657e0-b801-4e94-858d-116a405bf9f7-kube-api-access-qwpb9\") pod \"limitador-operator-controller-manager-5778bb4449-2n2lb\" (UID: \"1a2657e0-b801-4e94-858d-116a405bf9f7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:10:53.071291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.071259 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpb9\" (UniqueName: \"kubernetes.io/projected/1a2657e0-b801-4e94-858d-116a405bf9f7-kube-api-access-qwpb9\") pod \"limitador-operator-controller-manager-5778bb4449-2n2lb\" (UID: \"1a2657e0-b801-4e94-858d-116a405bf9f7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:10:53.110789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.110759 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:10:53.113572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.113545 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:10:53.124914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.124852 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpb9\" (UniqueName: \"kubernetes.io/projected/1a2657e0-b801-4e94-858d-116a405bf9f7-kube-api-access-qwpb9\") pod \"limitador-operator-controller-manager-5778bb4449-2n2lb\" (UID: \"1a2657e0-b801-4e94-858d-116a405bf9f7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:10:53.227890 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.227855 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kvghs\"" Nov 02 08:10:53.230873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.230850 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:10:53.592112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.592073 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" event={"ID":"e8a57647-59cb-49ba-a734-82752c851371","Type":"ContainerStarted","Data":"13c7d2f3ed735b48197aa183b4ff9067c42687fe1826d36f9e2ec7815c6d0ca3"} Nov 02 08:10:53.592576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.592268 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:10:53.594404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.594380 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" event={"ID":"0444d31b-6109-46c2-94ce-052305a30c0c","Type":"ContainerStarted","Data":"b1e90e1c0f59df9038f21217caedff54b588fb9cbf49b10442f3ff27ef3de059"} Nov 02 08:10:53.594570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.594557 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:10:53.612093 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.612063 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb"] Nov 02 08:10:53.615424 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:10:53.615395 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2657e0_b801_4e94_858d_116a405bf9f7.slice/crio-e2bfa15c083ffa5adb99c3d552c7068f6d1252dad052bb8f6f1bad17f476f7c7 WatchSource:0}: Error finding container e2bfa15c083ffa5adb99c3d552c7068f6d1252dad052bb8f6f1bad17f476f7c7: Status 404 returned error can't find the container with id e2bfa15c083ffa5adb99c3d552c7068f6d1252dad052bb8f6f1bad17f476f7c7 Nov 02 08:10:53.704152 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:53.704091 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" podStartSLOduration=2.8062598579999998 podStartE2EDuration="5.704077591s" totalImagesPullingTime="2.897817733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:53.702890399 +0000 UTC m=+300.793510812" watchObservedRunningTime="2026-11-02 08:10:53.704077591 +0000 UTC m=+300.794698010" Nov 02 08:10:54.600610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:54.600567 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" event={"ID":"1a2657e0-b801-4e94-858d-116a405bf9f7","Type":"ContainerStarted","Data":"e2bfa15c083ffa5adb99c3d552c7068f6d1252dad052bb8f6f1bad17f476f7c7"} Nov 02 08:10:54.918733 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:10:54.918652 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:10:57.616077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:57.616037 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" event={"ID":"1a2657e0-b801-4e94-858d-116a405bf9f7","Type":"ContainerStarted","Data":"091e0bf40cc1fbb004cc4bc4f9b0f9cceb2c58d0deaf53e57f4e981cea21aafe"} Nov 02 08:10:57.616563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:57.616164 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:10:57.724060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:10:57.724005 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" podStartSLOduration=2.704812715 podStartE2EDuration="5.723987808s" totalImagesPullingTime="3.019175093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:57.722126491 +0000 UTC m=+304.812746909" watchObservedRunningTime="2026-11-02 08:10:57.723987808 +0000 UTC m=+304.814608226" Nov 02 08:11:00.576005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:00.575971 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hqt9s" Nov 02 08:11:01.417884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.417835 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm"] Nov 02 08:11:01.422707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.422678 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.444191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.444124 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad0eb33-48f9-4d65-93ea-59f916a61471-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2nmjm\" (UID: \"cad0eb33-48f9-4d65-93ea-59f916a61471\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.444342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.444283 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxm2f\" (UniqueName: \"kubernetes.io/projected/cad0eb33-48f9-4d65-93ea-59f916a61471-kube-api-access-gxm2f\") pod \"kuadrant-operator-controller-manager-678d87766f-2nmjm\" (UID: \"cad0eb33-48f9-4d65-93ea-59f916a61471\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.468792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.468757 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm"] Nov 02 08:11:01.513028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.512992 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sqrf4\"" Nov 02 08:11:01.544926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.544878 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad0eb33-48f9-4d65-93ea-59f916a61471-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2nmjm\" (UID: \"cad0eb33-48f9-4d65-93ea-59f916a61471\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.545084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.545001 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxm2f\" (UniqueName: \"kubernetes.io/projected/cad0eb33-48f9-4d65-93ea-59f916a61471-kube-api-access-gxm2f\") pod \"kuadrant-operator-controller-manager-678d87766f-2nmjm\" (UID: \"cad0eb33-48f9-4d65-93ea-59f916a61471\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.545339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.545316 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad0eb33-48f9-4d65-93ea-59f916a61471-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2nmjm\" (UID: \"cad0eb33-48f9-4d65-93ea-59f916a61471\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.598642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.598602 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxm2f\" (UniqueName: \"kubernetes.io/projected/cad0eb33-48f9-4d65-93ea-59f916a61471-kube-api-access-gxm2f\") pod \"kuadrant-operator-controller-manager-678d87766f-2nmjm\" (UID: \"cad0eb33-48f9-4d65-93ea-59f916a61471\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:01.737314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:01.737271 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:02.151398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:02.151371 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm"] Nov 02 08:11:02.154333 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:11:02.154301 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad0eb33_48f9_4d65_93ea_59f916a61471.slice/crio-e3b4e5eaa2e747a76be458991bd112026ec729b922e5dea119a467dd9e3db09a WatchSource:0}: Error finding container e3b4e5eaa2e747a76be458991bd112026ec729b922e5dea119a467dd9e3db09a: Status 404 returned error can't find the container with id e3b4e5eaa2e747a76be458991bd112026ec729b922e5dea119a467dd9e3db09a Nov 02 08:11:02.157079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:02.157061 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:11:02.640749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:02.640714 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" event={"ID":"cad0eb33-48f9-4d65-93ea-59f916a61471","Type":"ContainerStarted","Data":"e3b4e5eaa2e747a76be458991bd112026ec729b922e5dea119a467dd9e3db09a"} Nov 02 08:11:04.603858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:04.603773 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-m25l7" Nov 02 08:11:04.603858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:04.603849 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x7d9h" Nov 02 08:11:08.623091 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:08.623059 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2n2lb" Nov 02 08:11:13.697845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:13.697802 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" event={"ID":"cad0eb33-48f9-4d65-93ea-59f916a61471","Type":"ContainerStarted","Data":"847ae30e95b69b605998e7ccfeae983ba08ef31469bab11df6be8a4796da4a36"} Nov 02 08:11:13.698249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:13.697935 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:13.899056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:13.898999 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" podStartSLOduration=2.080502545 podStartE2EDuration="12.898984244s" totalImagesPullingTime="10.818481699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:13.897815782 +0000 UTC m=+320.988436202" watchObservedRunningTime="2026-11-02 08:11:13.898984244 +0000 UTC m=+320.989604662" Nov 02 08:11:24.705058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:24.705027 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2nmjm" Nov 02 08:11:24.922023 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:11:24.921988 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:11:26.726168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.726118 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4"] Nov 02 08:11:26.733180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.733126 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.782641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.782581 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fb8f2\"" Nov 02 08:11:26.830248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.830209 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4"] Nov 02 08:11:26.845315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845281 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845342 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845363 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845455 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d37579c-29e7-4813-bf47-42bd05875256-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845483 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845507 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d37579c-29e7-4813-bf47-42bd05875256-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845527 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845658 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d37579c-29e7-4813-bf47-42bd05875256-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.845760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.845687 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvms\" (UniqueName: \"kubernetes.io/projected/1d37579c-29e7-4813-bf47-42bd05875256-kube-api-access-fmvms\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946383 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946429 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946463 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d37579c-29e7-4813-bf47-42bd05875256-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946502 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946535 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d37579c-29e7-4813-bf47-42bd05875256-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946565 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946602 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d37579c-29e7-4813-bf47-42bd05875256-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946626 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvms\" (UniqueName: \"kubernetes.io/projected/1d37579c-29e7-4813-bf47-42bd05875256-kube-api-access-fmvms\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.946705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.946681 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.947099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.947006 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.947189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.947101 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.947339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.947312 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.947339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.947332 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d37579c-29e7-4813-bf47-42bd05875256-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.947442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.947383 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.948901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.948877 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d37579c-29e7-4813-bf47-42bd05875256-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:26.949278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:26.949256 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d37579c-29e7-4813-bf47-42bd05875256-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:27.005670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.005595 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d37579c-29e7-4813-bf47-42bd05875256-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:27.006801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.006776 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvms\" (UniqueName: \"kubernetes.io/projected/1d37579c-29e7-4813-bf47-42bd05875256-kube-api-access-fmvms\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5vd6w4\" (UID: \"1d37579c-29e7-4813-bf47-42bd05875256\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:27.045099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.045067 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:27.472601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.472575 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4"] Nov 02 08:11:27.474951 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:11:27.474920 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d37579c_29e7_4813_bf47_42bd05875256.slice/crio-78bb99a61fcbb13ca9cbc93cf5b3bd94a9678a9ae63472efa663ee051ed6bf0a WatchSource:0}: Error finding container 78bb99a61fcbb13ca9cbc93cf5b3bd94a9678a9ae63472efa663ee051ed6bf0a: Status 404 returned error can't find the container with id 78bb99a61fcbb13ca9cbc93cf5b3bd94a9678a9ae63472efa663ee051ed6bf0a Nov 02 08:11:27.477169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.477113 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:11:27.477268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.477208 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:11:27.477268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.477241 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:11:27.771205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.771096 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" event={"ID":"1d37579c-29e7-4813-bf47-42bd05875256","Type":"ContainerStarted","Data":"78188dcc3a4f1b352d94a29e31a873f470102743edea3b5b19faa384d4289141"} Nov 02 08:11:27.771205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.771184 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" event={"ID":"1d37579c-29e7-4813-bf47-42bd05875256","Type":"ContainerStarted","Data":"78bb99a61fcbb13ca9cbc93cf5b3bd94a9678a9ae63472efa663ee051ed6bf0a"} Nov 02 08:11:27.880787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:27.880730 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" podStartSLOduration=1.8807150529999999 podStartE2EDuration="1.880715053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:27.87959942 +0000 UTC m=+334.970219840" watchObservedRunningTime="2026-11-02 08:11:27.880715053 +0000 UTC m=+334.971335476" Nov 02 08:11:28.045777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:28.045697 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:28.050570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:28.050543 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:28.775942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:28.775912 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:28.777009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:28.776988 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5vd6w4" Nov 02 08:11:30.347970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.347932 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zpdbb"] Nov 02 08:11:30.352180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.352160 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:30.377105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.377077 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mnk\" (UniqueName: \"kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk\") pod \"authorino-57f7f8659f-zpdbb\" (UID: \"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8\") " pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:30.403497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.403465 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zpdbb"] Nov 02 08:11:30.447489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.447453 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5grrd\"" Nov 02 08:11:30.478259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.478224 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mnk\" (UniqueName: \"kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk\") pod \"authorino-57f7f8659f-zpdbb\" (UID: \"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8\") " pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:30.531447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.531417 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mnk\" (UniqueName: \"kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk\") pod \"authorino-57f7f8659f-zpdbb\" (UID: \"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8\") " pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:30.675173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:30.675072 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:31.035707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:31.035678 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zpdbb"] Nov 02 08:11:31.038447 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:11:31.038403 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc41ab2d_e7e8_4ab1_861d_7d8177161ee8.slice/crio-05ecbf99d5c8a827669b8dc8e38153122236116b1651aa2d01730500630bcd28 WatchSource:0}: Error finding container 05ecbf99d5c8a827669b8dc8e38153122236116b1651aa2d01730500630bcd28: Status 404 returned error can't find the container with id 05ecbf99d5c8a827669b8dc8e38153122236116b1651aa2d01730500630bcd28 Nov 02 08:11:31.790495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:31.790441 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" event={"ID":"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8","Type":"ContainerStarted","Data":"05ecbf99d5c8a827669b8dc8e38153122236116b1651aa2d01730500630bcd28"} Nov 02 08:11:33.293545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.293509 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:11:33.297877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.297850 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.346155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.346117 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 08:11:33.348626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.348542 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:11:33.404633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.404596 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd5f2f01-bb6b-4133-a218-4507aeb2a055-config-file\") pod \"limitador-limitador-7c9b87d574-hfpsb\" (UID: \"fd5f2f01-bb6b-4133-a218-4507aeb2a055\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.404827 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.404720 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxn26\" (UniqueName: \"kubernetes.io/projected/fd5f2f01-bb6b-4133-a218-4507aeb2a055-kube-api-access-kxn26\") pod \"limitador-limitador-7c9b87d574-hfpsb\" (UID: \"fd5f2f01-bb6b-4133-a218-4507aeb2a055\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.441475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.441440 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p96n8\"" Nov 02 08:11:33.506326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.506290 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxn26\" (UniqueName: \"kubernetes.io/projected/fd5f2f01-bb6b-4133-a218-4507aeb2a055-kube-api-access-kxn26\") pod \"limitador-limitador-7c9b87d574-hfpsb\" (UID: \"fd5f2f01-bb6b-4133-a218-4507aeb2a055\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.506536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.506399 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd5f2f01-bb6b-4133-a218-4507aeb2a055-config-file\") pod \"limitador-limitador-7c9b87d574-hfpsb\" (UID: \"fd5f2f01-bb6b-4133-a218-4507aeb2a055\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.507119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.507096 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd5f2f01-bb6b-4133-a218-4507aeb2a055-config-file\") pod \"limitador-limitador-7c9b87d574-hfpsb\" (UID: \"fd5f2f01-bb6b-4133-a218-4507aeb2a055\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.558972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.558895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxn26\" (UniqueName: \"kubernetes.io/projected/fd5f2f01-bb6b-4133-a218-4507aeb2a055-kube-api-access-kxn26\") pod \"limitador-limitador-7c9b87d574-hfpsb\" (UID: \"fd5f2f01-bb6b-4133-a218-4507aeb2a055\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.585820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.585783 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:11:33.617113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.617078 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:33.990350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:33.990322 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:11:33.992804 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:11:33.992773 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5f2f01_bb6b_4133_a218_4507aeb2a055.slice/crio-96d6fbd0dd4484f20a5f500a372e268e4e392a367438abe4c7e76eba51f42d88 WatchSource:0}: Error finding container 96d6fbd0dd4484f20a5f500a372e268e4e392a367438abe4c7e76eba51f42d88: Status 404 returned error can't find the container with id 96d6fbd0dd4484f20a5f500a372e268e4e392a367438abe4c7e76eba51f42d88 Nov 02 08:11:34.807348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:34.807309 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" event={"ID":"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8","Type":"ContainerStarted","Data":"246ed1d5e02bd153e641326804b647cd40bcc0b0a92246c4165f27b1c22e50cb"} Nov 02 08:11:34.808575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:34.808550 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" event={"ID":"fd5f2f01-bb6b-4133-a218-4507aeb2a055","Type":"ContainerStarted","Data":"96d6fbd0dd4484f20a5f500a372e268e4e392a367438abe4c7e76eba51f42d88"} Nov 02 08:11:34.912335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:34.912272 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" podStartSLOduration=2.041593023 podStartE2EDuration="4.912253203s" totalImagesPullingTime="2.87066018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:34.911535543 +0000 UTC m=+342.002155961" watchObservedRunningTime="2026-11-02 08:11:34.912253203 +0000 UTC m=+342.002873625" Nov 02 08:11:36.820374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:36.820331 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" event={"ID":"fd5f2f01-bb6b-4133-a218-4507aeb2a055","Type":"ContainerStarted","Data":"36eaa10f7715d5eea757d37c2c1e7ab0b04cdd5086609571c0df51f7bc781b35"} Nov 02 08:11:36.820791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:36.820462 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:36.926434 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:36.926366 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" podStartSLOduration=1.911539793 podStartE2EDuration="3.926346704s" totalImagesPullingTime="2.014806911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:36.926067485 +0000 UTC m=+344.016687903" watchObservedRunningTime="2026-11-02 08:11:36.926346704 +0000 UTC m=+344.016967124" Nov 02 08:11:37.586659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.586621 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zpdbb"] Nov 02 08:11:37.586833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.586793 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" podUID="cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" containerName="authorino" containerID="cri-o://246ed1d5e02bd153e641326804b647cd40bcc0b0a92246c4165f27b1c22e50cb" gracePeriod=30 Nov 02 08:11:37.832668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.832640 5055 generic.go:356] "Generic (PLEG): container finished" podID="cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" containerID="246ed1d5e02bd153e641326804b647cd40bcc0b0a92246c4165f27b1c22e50cb" exitCode=0 Nov 02 08:11:37.833000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.832708 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" event={"ID":"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8","Type":"ContainerDied","Data":"246ed1d5e02bd153e641326804b647cd40bcc0b0a92246c4165f27b1c22e50cb"} Nov 02 08:11:37.852819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.852796 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:37.948323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.948291 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk\" (UniqueName: \"kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk\") pod \"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8\" (UID: \"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8\") " Nov 02 08:11:37.950386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:37.950356 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk" pod "cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" (UID: "cc41ab2d-e7e8-4ab1-861d-7d8177161ee8"). InnerVolumeSpecName "kube-api-access-c8mnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:11:38.049911 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:38.049864 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8mnk\" (UniqueName: \"kubernetes.io/projected/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8-kube-api-access-c8mnk\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:11:38.838932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:38.838901 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" Nov 02 08:11:38.839439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:38.838928 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zpdbb" event={"ID":"cc41ab2d-e7e8-4ab1-861d-7d8177161ee8","Type":"ContainerDied","Data":"05ecbf99d5c8a827669b8dc8e38153122236116b1651aa2d01730500630bcd28"} Nov 02 08:11:38.839439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:38.838977 5055 scope.go:122] "RemoveContainer" containerID="246ed1d5e02bd153e641326804b647cd40bcc0b0a92246c4165f27b1c22e50cb" Nov 02 08:11:38.988986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:38.988953 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zpdbb"] Nov 02 08:11:38.992285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:38.992248 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zpdbb"] Nov 02 08:11:39.091062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:39.090980 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" path="/var/lib/kubelet/pods/cc41ab2d-e7e8-4ab1-861d-7d8177161ee8/volumes" Nov 02 08:11:39.116050 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:39.115999 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" podUID="34d28ca7a91586006c95d24bde05028b" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:11:47.833807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:11:47.833773 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hfpsb" Nov 02 08:11:54.918511 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:11:54.918464 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:12:24.918364 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:12:24.918320 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:12:41.760602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.760568 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj"] Nov 02 08:12:41.761094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.760994 5055 cpu_manager.go:414] "containerMap: removing container" podUID="cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" containerName="authorino" Nov 02 08:12:41.761094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.761005 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" containerName="authorino" Nov 02 08:12:41.761094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.761079 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc41ab2d-e7e8-4ab1-861d-7d8177161ee8" containerName="authorino" Nov 02 08:12:41.764197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.764131 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:41.859235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.859198 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 08:12:41.859467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.859247 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 08:12:41.859613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.859250 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 08:12:41.859613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.859314 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 08:12:41.859613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.859321 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qpx4x\"" Nov 02 08:12:41.859613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.859423 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 08:12:41.861371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.861348 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj"] Nov 02 08:12:41.941409 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.941371 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:41.941587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.941519 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ba0863-68a4-4e22-895d-39cef92ac281-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:41.941587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.941566 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:41.941697 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:41.941618 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf99\" (UniqueName: \"kubernetes.io/projected/23ba0863-68a4-4e22-895d-39cef92ac281-kube-api-access-kbf99\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.042366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.042273 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ba0863-68a4-4e22-895d-39cef92ac281-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.042366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.042314 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.042606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.042443 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf99\" (UniqueName: \"kubernetes.io/projected/23ba0863-68a4-4e22-895d-39cef92ac281-kube-api-access-kbf99\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.042606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.042525 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.042722 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:12:42.042638 5055 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 08:12:42.042722 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:12:42.042709 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-proxy-source podName:23ba0863-68a4-4e22-895d-39cef92ac281 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:42.542690789 +0000 UTC m=+409.633311198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" (UID: "23ba0863-68a4-4e22-895d-39cef92ac281") : configmap "usage-tenancy-proxy-source" not found Nov 02 08:12:42.042967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.042941 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.044892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.044869 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ba0863-68a4-4e22-895d-39cef92ac281-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.095648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.095614 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf99\" (UniqueName: \"kubernetes.io/projected/23ba0863-68a4-4e22-895d-39cef92ac281-kube-api-access-kbf99\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.548948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.548891 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.549963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.549940 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ba0863-68a4-4e22-895d-39cef92ac281-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dxvdj\" (UID: \"23ba0863-68a4-4e22-895d-39cef92ac281\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:42.677367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:42.677329 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:43.040834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:43.040652 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj"] Nov 02 08:12:43.043622 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:12:43.043592 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ba0863_68a4_4e22_895d_39cef92ac281.slice/crio-4313b2227ccbed308e057500564a1c28fa7f51a90958834cafed36170de3cd8e WatchSource:0}: Error finding container 4313b2227ccbed308e057500564a1c28fa7f51a90958834cafed36170de3cd8e: Status 404 returned error can't find the container with id 4313b2227ccbed308e057500564a1c28fa7f51a90958834cafed36170de3cd8e Nov 02 08:12:43.164295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:43.164243 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" event={"ID":"23ba0863-68a4-4e22-895d-39cef92ac281","Type":"ContainerStarted","Data":"4313b2227ccbed308e057500564a1c28fa7f51a90958834cafed36170de3cd8e"} Nov 02 08:12:44.968209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:44.968170 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ccqh4"] Nov 02 08:12:44.975225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:44.975199 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.070765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.070727 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 08:12:45.070940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.070826 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 08:12:45.070989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.070968 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2grnc\"" Nov 02 08:12:45.072688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.072668 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ccqh4"] Nov 02 08:12:45.074282 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.074260 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19187067-159b-4ddf-a72f-fe3ccc9a180f-metrics-certs\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.074375 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.074293 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19187067-159b-4ddf-a72f-fe3ccc9a180f-cert\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.074485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.074457 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8r9\" (UniqueName: \"kubernetes.io/projected/19187067-159b-4ddf-a72f-fe3ccc9a180f-kube-api-access-rq8r9\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.174952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.174920 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19187067-159b-4ddf-a72f-fe3ccc9a180f-metrics-certs\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.175164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.174959 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19187067-159b-4ddf-a72f-fe3ccc9a180f-cert\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.175164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.174999 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8r9\" (UniqueName: \"kubernetes.io/projected/19187067-159b-4ddf-a72f-fe3ccc9a180f-kube-api-access-rq8r9\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.177533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.177506 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19187067-159b-4ddf-a72f-fe3ccc9a180f-cert\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.177647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.177539 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19187067-159b-4ddf-a72f-fe3ccc9a180f-metrics-certs\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.227872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.227799 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8r9\" (UniqueName: \"kubernetes.io/projected/19187067-159b-4ddf-a72f-fe3ccc9a180f-kube-api-access-rq8r9\") pod \"odh-model-controller-769875db56-ccqh4\" (UID: \"19187067-159b-4ddf-a72f-fe3ccc9a180f\") " pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.287901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.287860 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:45.647729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:45.647698 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ccqh4"] Nov 02 08:12:45.650329 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:12:45.650303 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19187067_159b_4ddf_a72f_fe3ccc9a180f.slice/crio-b85982756cbf2c0aa7fee029914b836d183bc4f87fe15fac6f20aeb30bbc80fa WatchSource:0}: Error finding container b85982756cbf2c0aa7fee029914b836d183bc4f87fe15fac6f20aeb30bbc80fa: Status 404 returned error can't find the container with id b85982756cbf2c0aa7fee029914b836d183bc4f87fe15fac6f20aeb30bbc80fa Nov 02 08:12:46.181191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:46.181132 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ccqh4" event={"ID":"19187067-159b-4ddf-a72f-fe3ccc9a180f","Type":"ContainerStarted","Data":"b85982756cbf2c0aa7fee029914b836d183bc4f87fe15fac6f20aeb30bbc80fa"} Nov 02 08:12:51.232606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:51.232570 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ccqh4" event={"ID":"19187067-159b-4ddf-a72f-fe3ccc9a180f","Type":"ContainerStarted","Data":"8b444b79bb59b124ceb6fef975011160a2451ba320bd4fec2e0885932e6f2d67"} Nov 02 08:12:51.233085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:51.232672 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:12:51.339696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:51.339636 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ccqh4" podStartSLOduration=2.716633041 podStartE2EDuration="7.339621983s" totalImagesPullingTime="4.622988942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:51.336994525 +0000 UTC m=+418.427614944" watchObservedRunningTime="2026-11-02 08:12:51.339621983 +0000 UTC m=+418.430242403" Nov 02 08:12:54.924887 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:12:54.924844 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:12:57.269739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:57.269699 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" event={"ID":"23ba0863-68a4-4e22-895d-39cef92ac281","Type":"ContainerStarted","Data":"13d97364661a90e32fabc42b86ec71180ac0ff8a39b408802eeda9424c1019eb"} Nov 02 08:12:57.270224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:57.269772 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:12:57.375529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:12:57.375429 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" podStartSLOduration=2.254548876 podStartE2EDuration="16.375408444s" totalImagesPullingTime="14.120859568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:57.374667611 +0000 UTC m=+424.465288032" watchObservedRunningTime="2026-11-02 08:12:57.375408444 +0000 UTC m=+424.466028863" Nov 02 08:13:02.240995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:02.240965 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ccqh4" Nov 02 08:13:08.280654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:08.280621 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dxvdj" Nov 02 08:13:19.426018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:19.425976 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:13:24.932344 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:13:24.932305 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:13:27.620181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:27.620125 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jvrbq"] Nov 02 08:13:27.871956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:27.871850 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jvrbq"] Nov 02 08:13:27.872184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:27.871973 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:27.949915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:27.949887 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5grrd\"" Nov 02 08:13:27.968285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:27.968249 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jvrbq"] Nov 02 08:13:27.968492 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:13:27.968471 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dhsgs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" podUID="88cfc302-b9b5-4081-afbc-a6942b141a9d" Nov 02 08:13:27.986852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:27.986818 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsgs\" (UniqueName: \"kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs\") pod \"authorino-6f6944bcdd-jvrbq\" (UID: \"88cfc302-b9b5-4081-afbc-a6942b141a9d\") " pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:28.001566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.001536 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b7b9b54fb-2b5qg"] Nov 02 08:13:28.031739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.031707 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:28.043249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.043215 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b7b9b54fb-2b5qg"] Nov 02 08:13:28.088280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.088237 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsgs\" (UniqueName: \"kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs\") pod \"authorino-6f6944bcdd-jvrbq\" (UID: \"88cfc302-b9b5-4081-afbc-a6942b141a9d\") " pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:28.141260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.141187 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsgs\" (UniqueName: \"kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs\") pod \"authorino-6f6944bcdd-jvrbq\" (UID: \"88cfc302-b9b5-4081-afbc-a6942b141a9d\") " pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:28.188954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.188909 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gr7\" (UniqueName: \"kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7\") pod \"authorino-5b7b9b54fb-2b5qg\" (UID: \"349ba43f-3743-46f3-b5db-6f472b9d9341\") " pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:28.290278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.290235 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gr7\" (UniqueName: \"kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7\") pod \"authorino-5b7b9b54fb-2b5qg\" (UID: \"349ba43f-3743-46f3-b5db-6f472b9d9341\") " pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:28.343458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.343420 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gr7\" (UniqueName: \"kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7\") pod \"authorino-5b7b9b54fb-2b5qg\" (UID: \"349ba43f-3743-46f3-b5db-6f472b9d9341\") " pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:28.362809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.362770 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:28.457455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.457417 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b7b9b54fb-2b5qg"] Nov 02 08:13:28.462943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.462914 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:28.473298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.473269 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dc679f98-zqbjw"] Nov 02 08:13:28.499217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.499184 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.509046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.509016 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:28.528659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.528619 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc679f98-zqbjw"] Nov 02 08:13:28.547582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.547545 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 08:13:28.592580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.592544 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs\" (UniqueName: \"kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs\") pod \"88cfc302-b9b5-4081-afbc-a6942b141a9d\" (UID: \"88cfc302-b9b5-4081-afbc-a6942b141a9d\") " Nov 02 08:13:28.592770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.592711 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert\") pod \"authorino-dc679f98-zqbjw\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.592770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.592749 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5csg\" (UniqueName: \"kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg\") pod \"authorino-dc679f98-zqbjw\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.594831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.594801 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs" pod "88cfc302-b9b5-4081-afbc-a6942b141a9d" (UID: "88cfc302-b9b5-4081-afbc-a6942b141a9d"). InnerVolumeSpecName "kube-api-access-dhsgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:13:28.693683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.693594 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert\") pod \"authorino-dc679f98-zqbjw\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.693683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.693657 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5csg\" (UniqueName: \"kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg\") pod \"authorino-dc679f98-zqbjw\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.694097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.693709 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsgs\" (UniqueName: \"kubernetes.io/projected/88cfc302-b9b5-4081-afbc-a6942b141a9d-kube-api-access-dhsgs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:13:28.696092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.696066 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert\") pod \"authorino-dc679f98-zqbjw\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.736561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.736531 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b7b9b54fb-2b5qg"] Nov 02 08:13:28.738770 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:13:28.738740 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod349ba43f_3743_46f3_b5db_6f472b9d9341.slice/crio-67b954fe9361e2e62eeb8f58ed9a3d78cdd4970e34924dc93476059dbe2c1695 WatchSource:0}: Error finding container 67b954fe9361e2e62eeb8f58ed9a3d78cdd4970e34924dc93476059dbe2c1695: Status 404 returned error can't find the container with id 67b954fe9361e2e62eeb8f58ed9a3d78cdd4970e34924dc93476059dbe2c1695 Nov 02 08:13:28.746113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.746091 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5csg\" (UniqueName: \"kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg\") pod \"authorino-dc679f98-zqbjw\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:28.832447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:28.832410 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:13:29.226518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:29.226491 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc679f98-zqbjw"] Nov 02 08:13:29.228566 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:13:29.228531 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9db0b7d_f12d_4cd6_b387_93a5396bdb2c.slice/crio-477a36414d694cffdad93854ba07996b40111f52c1e44fd02912a38c515c739d WatchSource:0}: Error finding container 477a36414d694cffdad93854ba07996b40111f52c1e44fd02912a38c515c739d: Status 404 returned error can't find the container with id 477a36414d694cffdad93854ba07996b40111f52c1e44fd02912a38c515c739d Nov 02 08:13:29.468712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:29.468673 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" event={"ID":"349ba43f-3743-46f3-b5db-6f472b9d9341","Type":"ContainerStarted","Data":"67b954fe9361e2e62eeb8f58ed9a3d78cdd4970e34924dc93476059dbe2c1695"} Nov 02 08:13:29.469940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:29.469915 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jvrbq" Nov 02 08:13:29.470079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:29.469920 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc679f98-zqbjw" event={"ID":"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c","Type":"ContainerStarted","Data":"477a36414d694cffdad93854ba07996b40111f52c1e44fd02912a38c515c739d"} Nov 02 08:13:29.675114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:29.675078 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jvrbq"] Nov 02 08:13:29.963595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:29.676763 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jvrbq"] Nov 02 08:13:30.476596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:30.476558 5055 generic.go:356] "Generic (PLEG): container finished" podID="d7de352b-640f-49b5-873b-1f8873d031b1" containerID="70db185696ddaba8f5d5bec00676ed4fc818b4fb194cd70d346420608cb8f32b" exitCode=0 Nov 02 08:13:30.476786 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:30.476630 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" event={"ID":"d7de352b-640f-49b5-873b-1f8873d031b1","Type":"ContainerDied","Data":"70db185696ddaba8f5d5bec00676ed4fc818b4fb194cd70d346420608cb8f32b"} Nov 02 08:13:30.477103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:30.477073 5055 scope.go:122] "RemoveContainer" containerID="70db185696ddaba8f5d5bec00676ed4fc818b4fb194cd70d346420608cb8f32b" Nov 02 08:13:30.486511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:30.486471 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:13:30.486568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:30.486555 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:13:30.486602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:30.486592 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:13:31.109696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.109612 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88cfc302-b9b5-4081-afbc-a6942b141a9d" path="/var/lib/kubelet/pods/88cfc302-b9b5-4081-afbc-a6942b141a9d/volumes" Nov 02 08:13:31.506975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.506931 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" event={"ID":"d7de352b-640f-49b5-873b-1f8873d031b1","Type":"ContainerStarted","Data":"5c4092330116ff2702695048cc872bba003b9c45809f05ee963ce6bea7b0a09b"} Nov 02 08:13:31.509449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.509413 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc679f98-zqbjw" event={"ID":"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c","Type":"ContainerStarted","Data":"7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de"} Nov 02 08:13:31.512322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.512295 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" event={"ID":"349ba43f-3743-46f3-b5db-6f472b9d9341","Type":"ContainerStarted","Data":"c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f"} Nov 02 08:13:31.512470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.512384 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" podUID="349ba43f-3743-46f3-b5db-6f472b9d9341" containerName="authorino" containerID="cri-o://c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f" gracePeriod=30 Nov 02 08:13:31.727302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.725174 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" podStartSLOduration=2.979000938 podStartE2EDuration="4.725121598s" totalImagesPullingTime="1.74612066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:31.725086304 +0000 UTC m=+458.815706736" watchObservedRunningTime="2026-11-02 08:13:31.725121598 +0000 UTC m=+458.815742019" Nov 02 08:13:31.830446 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.830383 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-dc679f98-zqbjw" podStartSLOduration=2.3460639690000002 podStartE2EDuration="3.830366303s" totalImagesPullingTime="1.484302334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:31.829076234 +0000 UTC m=+458.919696653" watchObservedRunningTime="2026-11-02 08:13:31.830366303 +0000 UTC m=+458.920986723" Nov 02 08:13:31.871957 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.871923 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:31.994989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.982535 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:13:31.994989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.982676 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tskjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 08:13:31.994989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:31.982725 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" podUID="d7de352b-640f-49b5-873b-1f8873d031b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 08:13:32.029210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.029124 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7\" (UniqueName: \"kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7\") pod \"349ba43f-3743-46f3-b5db-6f472b9d9341\" (UID: \"349ba43f-3743-46f3-b5db-6f472b9d9341\") " Nov 02 08:13:32.031300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.031267 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7" pod "349ba43f-3743-46f3-b5db-6f472b9d9341" (UID: "349ba43f-3743-46f3-b5db-6f472b9d9341"). InnerVolumeSpecName "kube-api-access-86gr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:13:32.131721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.131162 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86gr7\" (UniqueName: \"kubernetes.io/projected/349ba43f-3743-46f3-b5db-6f472b9d9341-kube-api-access-86gr7\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:13:32.519563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.519519 5055 generic.go:356] "Generic (PLEG): container finished" podID="349ba43f-3743-46f3-b5db-6f472b9d9341" containerID="c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f" exitCode=0 Nov 02 08:13:32.519943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.519901 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" event={"ID":"349ba43f-3743-46f3-b5db-6f472b9d9341","Type":"ContainerDied","Data":"c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f"} Nov 02 08:13:32.520113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.519967 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" event={"ID":"349ba43f-3743-46f3-b5db-6f472b9d9341","Type":"ContainerDied","Data":"67b954fe9361e2e62eeb8f58ed9a3d78cdd4970e34924dc93476059dbe2c1695"} Nov 02 08:13:32.520113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.519996 5055 scope.go:122] "RemoveContainer" containerID="c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f" Nov 02 08:13:32.520113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.520070 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b7b9b54fb-2b5qg" Nov 02 08:13:32.529758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.529729 5055 scope.go:122] "RemoveContainer" containerID="c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f" Nov 02 08:13:32.530116 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:13:32.530093 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f\": container with ID starting with c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f not found: ID does not exist" containerID="c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f" Nov 02 08:13:32.530226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.530126 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f"} err="failed to get container status \"c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f\": rpc error: code = NotFound desc = could not find container \"c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f\": container with ID starting with c2e86d00bca11aa0f75b5136c8f1285a00729aabeb1a4c0f26b61e2b6a89b74f not found: ID does not exist" Nov 02 08:13:32.671048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.671015 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b7b9b54fb-2b5qg"] Nov 02 08:13:32.674658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.674631 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b7b9b54fb-2b5qg"] Nov 02 08:13:32.989175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.986594 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tskjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 08:13:32.989175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:32.986663 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" podUID="d7de352b-640f-49b5-873b-1f8873d031b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 08:13:33.111501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:33.111461 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349ba43f-3743-46f3-b5db-6f472b9d9341" path="/var/lib/kubelet/pods/349ba43f-3743-46f3-b5db-6f472b9d9341/volumes" Nov 02 08:13:34.005297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:34.005261 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tskjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 08:13:34.005768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:34.005743 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" podUID="d7de352b-640f-49b5-873b-1f8873d031b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 08:13:34.985166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:34.981588 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tskjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 08:13:34.985166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:34.981654 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" podUID="d7de352b-640f-49b5-873b-1f8873d031b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 08:13:35.997174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:35.996623 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tskjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 08:13:35.997174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:35.996688 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" podUID="d7de352b-640f-49b5-873b-1f8873d031b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 08:13:36.985235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:36.985201 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:13:36.985530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:36.985506 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:13:36.986359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:13:36.986334 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tskjl" Nov 02 08:13:54.919600 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:13:54.919557 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:14:24.919132 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:14:24.919047 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:14:54.919248 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:14:54.919204 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:15:24.919534 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:15:24.919499 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:15:39.613976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:39.613891 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:15:41.422222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.422181 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg"] Nov 02 08:15:41.422859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.422840 5055 cpu_manager.go:414] "containerMap: removing container" podUID="349ba43f-3743-46f3-b5db-6f472b9d9341" containerName="authorino" Nov 02 08:15:41.422932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.422861 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349ba43f-3743-46f3-b5db-6f472b9d9341" containerName="authorino" Nov 02 08:15:41.422967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.422943 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="349ba43f-3743-46f3-b5db-6f472b9d9341" containerName="authorino" Nov 02 08:15:41.425864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.425845 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.476118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.476089 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 08:15:41.478698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.478678 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 08:15:41.522269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.522239 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6gfs\"" Nov 02 08:15:41.524134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.524110 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg"] Nov 02 08:15:41.524910 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.524886 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 08:15:41.542566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.542529 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.542704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.542572 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.542704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.542593 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702fd86a-7fd7-496c-ae16-105cc49fd781-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.542781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.542699 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.542781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.542758 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8kb\" (UniqueName: \"kubernetes.io/projected/702fd86a-7fd7-496c-ae16-105cc49fd781-kube-api-access-5l8kb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.542856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.542812 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644003 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8kb\" (UniqueName: \"kubernetes.io/projected/702fd86a-7fd7-496c-ae16-105cc49fd781-kube-api-access-5l8kb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644050 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644128 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644185 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644211 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702fd86a-7fd7-496c-ae16-105cc49fd781-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644244 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644612 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644590 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644624 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.644687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.644659 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.646527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.646506 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702fd86a-7fd7-496c-ae16-105cc49fd781-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.646845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.646826 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702fd86a-7fd7-496c-ae16-105cc49fd781-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.698473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.698396 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8kb\" (UniqueName: \"kubernetes.io/projected/702fd86a-7fd7-496c-ae16-105cc49fd781-kube-api-access-5l8kb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2fdsg\" (UID: \"702fd86a-7fd7-496c-ae16-105cc49fd781\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:41.738733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:41.738693 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:42.111453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:42.111421 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg"] Nov 02 08:15:42.113535 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:15:42.113491 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702fd86a_7fd7_496c_ae16_105cc49fd781.slice/crio-85852429d2d666f43e2bde37021af3c57744c492142f9db2105cb3079634e4a7 WatchSource:0}: Error finding container 85852429d2d666f43e2bde37021af3c57744c492142f9db2105cb3079634e4a7: Status 404 returned error can't find the container with id 85852429d2d666f43e2bde37021af3c57744c492142f9db2105cb3079634e4a7 Nov 02 08:15:42.173340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:42.173299 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" event={"ID":"702fd86a-7fd7-496c-ae16-105cc49fd781","Type":"ContainerStarted","Data":"85852429d2d666f43e2bde37021af3c57744c492142f9db2105cb3079634e4a7"} Nov 02 08:15:44.188125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:44.188017 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" event={"ID":"702fd86a-7fd7-496c-ae16-105cc49fd781","Type":"ContainerStarted","Data":"a6c6b27784e884aab8c314833116ef68d75b7b50701b1221c7156f07a2895f13"} Nov 02 08:15:44.299169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:44.298858 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" podStartSLOduration=1.6933732190000002 podStartE2EDuration="3.29883534s" totalImagesPullingTime="1.605462121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:44.298278848 +0000 UTC m=+591.388899267" watchObservedRunningTime="2026-11-02 08:15:44.29883534 +0000 UTC m=+591.389455760" Nov 02 08:15:44.739810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:44.739769 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:45.170412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.170335 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv"] Nov 02 08:15:45.174187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.174163 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.268622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.268593 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 08:15:45.270791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.270769 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv"] Nov 02 08:15:45.285490 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.285461 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.285620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.285507 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqh5f\" (UniqueName: \"kubernetes.io/projected/71541873-6433-439a-86df-75e6f31f5c2c-kube-api-access-lqh5f\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.285620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.285529 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.285719 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.285690 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.285756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.285742 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71541873-6433-439a-86df-75e6f31f5c2c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.285869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.285849 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386323 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386368 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71541873-6433-439a-86df-75e6f31f5c2c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386407 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386443 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386482 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqh5f\" (UniqueName: \"kubernetes.io/projected/71541873-6433-439a-86df-75e6f31f5c2c-kube-api-access-lqh5f\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386549 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386801 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386825 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.386943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.386887 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.388866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.388836 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71541873-6433-439a-86df-75e6f31f5c2c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.389064 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.389048 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71541873-6433-439a-86df-75e6f31f5c2c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.440278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.440205 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqh5f\" (UniqueName: \"kubernetes.io/projected/71541873-6433-439a-86df-75e6f31f5c2c-kube-api-access-lqh5f\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w5vkv\" (UID: \"71541873-6433-439a-86df-75e6f31f5c2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.487243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.487199 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:45.855003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:45.854975 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv"] Nov 02 08:15:45.856844 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:15:45.856806 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71541873_6433_439a_86df_75e6f31f5c2c.slice/crio-03880005be252ebcd2ed479cdb8db1951a1f1ac4b58cec543b5a3d0bfaa7ca8f WatchSource:0}: Error finding container 03880005be252ebcd2ed479cdb8db1951a1f1ac4b58cec543b5a3d0bfaa7ca8f: Status 404 returned error can't find the container with id 03880005be252ebcd2ed479cdb8db1951a1f1ac4b58cec543b5a3d0bfaa7ca8f Nov 02 08:15:46.204779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:46.204743 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" event={"ID":"71541873-6433-439a-86df-75e6f31f5c2c","Type":"ContainerStarted","Data":"03880005be252ebcd2ed479cdb8db1951a1f1ac4b58cec543b5a3d0bfaa7ca8f"} Nov 02 08:15:47.211684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:47.211643 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" event={"ID":"71541873-6433-439a-86df-75e6f31f5c2c","Type":"ContainerStarted","Data":"ecffd76792cf74e8359de58439da48f00b1a722fb3fe98c0312a56f539edc9b9"} Nov 02 08:15:47.320465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:47.320376 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" podStartSLOduration=1.6358182559999999 podStartE2EDuration="2.320358861s" totalImagesPullingTime="684.540605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:47.31940746 +0000 UTC m=+594.410027878" watchObservedRunningTime="2026-11-02 08:15:47.320358861 +0000 UTC m=+594.410979280" Nov 02 08:15:47.488015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:47.487919 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:49.167788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:49.167748 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:15:51.739312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:51.739270 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:51.751600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:51.751574 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:52.249963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:52.249931 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2fdsg" Nov 02 08:15:53.146253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:53.146207 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:15:53.150675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:53.150645 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:15:54.919115 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:15:54.919074 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:15:55.487753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.487707 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:55.500282 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.500253 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:55.836742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.836661 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lhp52"] Nov 02 08:15:55.840720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.840670 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:55.891240 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.891211 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 08:15:55.897530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.897501 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lhp52"] Nov 02 08:15:55.992076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.992030 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125943c9-4353-4684-80cd-f53ffe83a574-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:55.992571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.992168 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:55.992571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.992219 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-home\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:55.992571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.992474 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:55.992571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.992512 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:55.992571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:55.992542 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2tl\" (UniqueName: \"kubernetes.io/projected/125943c9-4353-4684-80cd-f53ffe83a574-kube-api-access-hw2tl\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.093977 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-home\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094037 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094168 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094203 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2tl\" (UniqueName: \"kubernetes.io/projected/125943c9-4353-4684-80cd-f53ffe83a574-kube-api-access-hw2tl\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094239 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125943c9-4353-4684-80cd-f53ffe83a574-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094327 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094580 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-home\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094627 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.094711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.094654 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.096454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.096422 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125943c9-4353-4684-80cd-f53ffe83a574-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.096709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.096692 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125943c9-4353-4684-80cd-f53ffe83a574-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.147662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.147634 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2tl\" (UniqueName: \"kubernetes.io/projected/125943c9-4353-4684-80cd-f53ffe83a574-kube-api-access-hw2tl\") pod \"echo-simulator-kserve-64b6f8f96c-lhp52\" (UID: \"125943c9-4353-4684-80cd-f53ffe83a574\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.156492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.156467 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:15:56.273301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.273269 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w5vkv" Nov 02 08:15:56.526571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:56.526543 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lhp52"] Nov 02 08:15:56.529206 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:15:56.529174 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125943c9_4353_4684_80cd_f53ffe83a574.slice/crio-74c5ae41d695dc356f600f9eabd079157700828a3c88a62f3c9f98a9fe728bc5 WatchSource:0}: Error finding container 74c5ae41d695dc356f600f9eabd079157700828a3c88a62f3c9f98a9fe728bc5: Status 404 returned error can't find the container with id 74c5ae41d695dc356f600f9eabd079157700828a3c88a62f3c9f98a9fe728bc5 Nov 02 08:15:57.273835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:57.273798 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" event={"ID":"125943c9-4353-4684-80cd-f53ffe83a574","Type":"ContainerStarted","Data":"74c5ae41d695dc356f600f9eabd079157700828a3c88a62f3c9f98a9fe728bc5"} Nov 02 08:15:59.286837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:59.286798 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" event={"ID":"125943c9-4353-4684-80cd-f53ffe83a574","Type":"ContainerStarted","Data":"431526cd0150fbf6d7132116931d85f56284e9a86af60c3375a47b1fc1f1399e"} Nov 02 08:15:59.393880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:59.393772 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" podStartSLOduration=1.8713628020000002 podStartE2EDuration="4.393756806s" totalImagesPullingTime="2.522394004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:59.392094013 +0000 UTC m=+606.482714456" watchObservedRunningTime="2026-11-02 08:15:59.393756806 +0000 UTC m=+606.484377222" Nov 02 08:15:59.595161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:15:59.594497 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:16:00.156725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:00.156691 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:16:03.583764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.583724 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k"] Nov 02 08:16:03.587775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.587747 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.675568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.675524 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.675754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.675577 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d13bf2a-d174-4b83-b363-ed42f963cff7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.675754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.675710 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.675754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.675752 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.675879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.675854 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.675923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.675893 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrn5\" (UniqueName: \"kubernetes.io/projected/5d13bf2a-d174-4b83-b363-ed42f963cff7-kube-api-access-shrn5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.681918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.681893 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 08:16:03.683986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.683965 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k"] Nov 02 08:16:03.776946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.776906 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.776946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.776947 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777017 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777183 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrn5\" (UniqueName: \"kubernetes.io/projected/5d13bf2a-d174-4b83-b363-ed42f963cff7-kube-api-access-shrn5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777230 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777264 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d13bf2a-d174-4b83-b363-ed42f963cff7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777402 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777415 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.777655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.777625 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.779485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.779459 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d13bf2a-d174-4b83-b363-ed42f963cff7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.780257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.780235 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d13bf2a-d174-4b83-b363-ed42f963cff7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.833359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.833326 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrn5\" (UniqueName: \"kubernetes.io/projected/5d13bf2a-d174-4b83-b363-ed42f963cff7-kube-api-access-shrn5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k\" (UID: \"5d13bf2a-d174-4b83-b363-ed42f963cff7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:03.900326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:03.900238 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:04.273580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:04.273553 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k"] Nov 02 08:16:04.275909 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:16:04.275865 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d13bf2a_d174_4b83_b363_ed42f963cff7.slice/crio-8c6ea8250edbff7d01b7f30b72f9de1f2df33b45f70d124c21440aa0c2a72036 WatchSource:0}: Error finding container 8c6ea8250edbff7d01b7f30b72f9de1f2df33b45f70d124c21440aa0c2a72036: Status 404 returned error can't find the container with id 8c6ea8250edbff7d01b7f30b72f9de1f2df33b45f70d124c21440aa0c2a72036 Nov 02 08:16:04.277498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:04.277480 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:16:04.314651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:04.314613 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" event={"ID":"5d13bf2a-d174-4b83-b363-ed42f963cff7","Type":"ContainerStarted","Data":"8c6ea8250edbff7d01b7f30b72f9de1f2df33b45f70d124c21440aa0c2a72036"} Nov 02 08:16:05.320932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:05.320853 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" event={"ID":"5d13bf2a-d174-4b83-b363-ed42f963cff7","Type":"ContainerStarted","Data":"f8b530dc8db87c60eb5c8ec7c1d662cfe14ac3f61ee3d1bf71ace445e9b57097"} Nov 02 08:16:05.432920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:05.432866 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" podStartSLOduration=1.746797923 podStartE2EDuration="2.432852868s" totalImagesPullingTime="686.054945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:05.430491311 +0000 UTC m=+612.521111730" watchObservedRunningTime="2026-11-02 08:16:05.432852868 +0000 UTC m=+612.523473284" Nov 02 08:16:05.900897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:05.900865 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:06.157538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:06.157434 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:16:06.169947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:06.169919 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:16:06.337303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:06.337274 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lhp52" Nov 02 08:16:13.901472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:13.901431 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:13.913943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:13.913917 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:14.385641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:14.385606 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" Nov 02 08:16:15.692782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:15.692739 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:16:20.592693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.592652 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59"] Nov 02 08:16:20.599564 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.599540 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.685398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.685347 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59"] Nov 02 08:16:20.761424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761382 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761451 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761599 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqcbc\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761640 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761737 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761760 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761824 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761864 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.761975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.761908 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863377 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqcbc\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863416 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863444 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863460 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863489 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863526 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863561 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863615 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.863813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863662 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.864116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.863995 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.864116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.864035 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.864284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.864171 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.864284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.864252 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.864460 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.864439 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.866058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.866030 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.866223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.866204 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.918879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.918847 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqcbc\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:20.919032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:20.918915 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-95j59\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:21.212055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:21.212015 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:21.628474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:21.628449 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59"] Nov 02 08:16:21.630251 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:16:21.630224 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e83a04b_acfb_4583_b44c_a52d18e16588.slice/crio-76c5a4bbbe00e4f21aa4de6cd52b94c4812ad56f01fc3cc652fa8568dc1c7924 WatchSource:0}: Error finding container 76c5a4bbbe00e4f21aa4de6cd52b94c4812ad56f01fc3cc652fa8568dc1c7924: Status 404 returned error can't find the container with id 76c5a4bbbe00e4f21aa4de6cd52b94c4812ad56f01fc3cc652fa8568dc1c7924 Nov 02 08:16:21.632549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:21.632512 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:16:21.632629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:21.632579 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:16:21.632629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:21.632610 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:16:22.420566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:22.420519 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" event={"ID":"3e83a04b-acfb-4583-b44c-a52d18e16588","Type":"ContainerStarted","Data":"7ab7198eb76c1004cd558996a7146d09da214e7e9c84a763cd787334be064c3d"} Nov 02 08:16:22.420778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:22.420593 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" event={"ID":"3e83a04b-acfb-4583-b44c-a52d18e16588","Type":"ContainerStarted","Data":"76c5a4bbbe00e4f21aa4de6cd52b94c4812ad56f01fc3cc652fa8568dc1c7924"} Nov 02 08:16:22.625548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:22.625477 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podStartSLOduration=2.625457097 podStartE2EDuration="2.625457097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:22.577979696 +0000 UTC m=+629.668600115" watchObservedRunningTime="2026-11-02 08:16:22.625457097 +0000 UTC m=+629.716077518" Nov 02 08:16:23.213059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:23.213024 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:23.214721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:23.214673 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-95j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 08:16:23.214889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:23.214761 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 08:16:24.212670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:24.212632 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-95j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 08:16:24.212863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:24.212686 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 08:16:24.929612 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:16:24.929573 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:16:25.212669 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:25.212633 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-95j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 08:16:25.212850 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:25.212688 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 08:16:26.213388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:26.213350 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-95j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 08:16:26.213808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:26.213405 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 08:16:27.212595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:27.212553 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-95j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 08:16:27.212756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:27.212612 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 08:16:28.212489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:28.212447 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-95j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 08:16:28.212909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:28.212512 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 08:16:29.942640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:29.942605 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:29.943121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:29.943090 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:29.943817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:29.943798 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:16:32.822446 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:32.822405 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69dbd95c57-6q7xh"] Nov 02 08:16:32.829513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:32.829487 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:32.881135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:32.881099 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69dbd95c57-6q7xh"] Nov 02 08:16:32.904063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:32.904015 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/915d8632-259a-4284-b10e-46ede9f3d67a-tls-cert\") pod \"authorino-69dbd95c57-6q7xh\" (UID: \"915d8632-259a-4284-b10e-46ede9f3d67a\") " pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:32.904063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:32.904072 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65kf\" (UniqueName: \"kubernetes.io/projected/915d8632-259a-4284-b10e-46ede9f3d67a-kube-api-access-h65kf\") pod \"authorino-69dbd95c57-6q7xh\" (UID: \"915d8632-259a-4284-b10e-46ede9f3d67a\") " pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:33.005737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:33.005688 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/915d8632-259a-4284-b10e-46ede9f3d67a-tls-cert\") pod \"authorino-69dbd95c57-6q7xh\" (UID: \"915d8632-259a-4284-b10e-46ede9f3d67a\") " pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:33.005737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:33.005747 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65kf\" (UniqueName: \"kubernetes.io/projected/915d8632-259a-4284-b10e-46ede9f3d67a-kube-api-access-h65kf\") pod \"authorino-69dbd95c57-6q7xh\" (UID: \"915d8632-259a-4284-b10e-46ede9f3d67a\") " pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:33.008634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:33.008598 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/915d8632-259a-4284-b10e-46ede9f3d67a-tls-cert\") pod \"authorino-69dbd95c57-6q7xh\" (UID: \"915d8632-259a-4284-b10e-46ede9f3d67a\") " pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:33.061235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:33.061197 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65kf\" (UniqueName: \"kubernetes.io/projected/915d8632-259a-4284-b10e-46ede9f3d67a-kube-api-access-h65kf\") pod \"authorino-69dbd95c57-6q7xh\" (UID: \"915d8632-259a-4284-b10e-46ede9f3d67a\") " pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:33.143071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:33.142975 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69dbd95c57-6q7xh" Nov 02 08:16:33.509417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:33.509286 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69dbd95c57-6q7xh"] Nov 02 08:16:33.512572 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:16:33.512541 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod915d8632_259a_4284_b10e_46ede9f3d67a.slice/crio-2c4854cf84fedb2e6be3a1d8fe5e4ce2bd9609945c168efd10696c92d1dddfef WatchSource:0}: Error finding container 2c4854cf84fedb2e6be3a1d8fe5e4ce2bd9609945c168efd10696c92d1dddfef: Status 404 returned error can't find the container with id 2c4854cf84fedb2e6be3a1d8fe5e4ce2bd9609945c168efd10696c92d1dddfef Nov 02 08:16:34.492903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:34.492862 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69dbd95c57-6q7xh" event={"ID":"915d8632-259a-4284-b10e-46ede9f3d67a","Type":"ContainerStarted","Data":"2c4854cf84fedb2e6be3a1d8fe5e4ce2bd9609945c168efd10696c92d1dddfef"} Nov 02 08:16:35.500190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:35.500153 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69dbd95c57-6q7xh" event={"ID":"915d8632-259a-4284-b10e-46ede9f3d67a","Type":"ContainerStarted","Data":"a34e21c13b025c3d66477de4ee74863104d9dbfea89158603c57fe29f5dcf52d"} Nov 02 08:16:35.624105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:35.624048 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69dbd95c57-6q7xh" podStartSLOduration=1.969471048 podStartE2EDuration="3.624031851s" totalImagesPullingTime="1.654560803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:35.621914667 +0000 UTC m=+642.712535086" watchObservedRunningTime="2026-11-02 08:16:35.624031851 +0000 UTC m=+642.714652269" Nov 02 08:16:35.695984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:35.695933 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc679f98-zqbjw"] Nov 02 08:16:35.696213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:35.696179 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-dc679f98-zqbjw" podUID="a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" containerName="authorino" containerID="cri-o://7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de" gracePeriod=30 Nov 02 08:16:35.958748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:35.958722 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:16:36.037444 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.037408 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert\" (UniqueName: \"kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert\") pod \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " Nov 02 08:16:36.037654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.037488 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg\" (UniqueName: \"kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg\") pod \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\" (UID: \"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c\") " Nov 02 08:16:36.039574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.039543 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg" pod "a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" (UID: "a9db0b7d-f12d-4cd6-b387-93a5396bdb2c"). InnerVolumeSpecName "kube-api-access-x5csg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:16:36.049961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.049925 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert" pod "a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" (UID: "a9db0b7d-f12d-4cd6-b387-93a5396bdb2c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:16:36.138523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.138441 5055 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-tls-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:16:36.138668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.138585 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5csg\" (UniqueName: \"kubernetes.io/projected/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c-kube-api-access-x5csg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:16:36.508452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.508408 5055 generic.go:356] "Generic (PLEG): container finished" podID="a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" containerID="7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de" exitCode=0 Nov 02 08:16:36.508452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.508451 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc679f98-zqbjw" Nov 02 08:16:36.509009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.508487 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc679f98-zqbjw" event={"ID":"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c","Type":"ContainerDied","Data":"7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de"} Nov 02 08:16:36.509009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.508531 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc679f98-zqbjw" event={"ID":"a9db0b7d-f12d-4cd6-b387-93a5396bdb2c","Type":"ContainerDied","Data":"477a36414d694cffdad93854ba07996b40111f52c1e44fd02912a38c515c739d"} Nov 02 08:16:36.509009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.508557 5055 scope.go:122] "RemoveContainer" containerID="7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de" Nov 02 08:16:36.523448 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.523421 5055 scope.go:122] "RemoveContainer" containerID="7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de" Nov 02 08:16:36.523848 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:16:36.523824 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de\": container with ID starting with 7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de not found: ID does not exist" containerID="7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de" Nov 02 08:16:36.523951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.523857 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de"} err="failed to get container status \"7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de\": rpc error: code = NotFound desc = could not find container \"7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de\": container with ID starting with 7d12659523134ac205d3f491e817654edb4843ec2b98ecdecb9abfccdcd818de not found: ID does not exist" Nov 02 08:16:36.662503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.662462 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc679f98-zqbjw"] Nov 02 08:16:36.666913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:36.666879 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-dc679f98-zqbjw"] Nov 02 08:16:37.091735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:16:37.091696 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" path="/var/lib/kubelet/pods/a9db0b7d-f12d-4cd6-b387-93a5396bdb2c/volumes" Nov 02 08:16:54.919668 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:16:54.919625 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:17:02.700086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.700044 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm"] Nov 02 08:17:02.700563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.700531 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" containerName="authorino" Nov 02 08:17:02.700563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.700544 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" containerName="authorino" Nov 02 08:17:02.700642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.700625 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9db0b7d-f12d-4cd6-b387-93a5396bdb2c" containerName="authorino" Nov 02 08:17:02.705459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.705431 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.763595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.763557 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2b62c067-openshift-default-dockercfg-hblqx\"" Nov 02 08:17:02.812296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.812256 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm"] Nov 02 08:17:02.813949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.813916 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814015 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814094 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814170 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814219 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814243 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814344 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814384 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.814425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.814400 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jfq\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.915855 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.915813 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.915855 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.915858 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.915899 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.915946 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.915974 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jfq\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916026 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916118 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916178 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916203 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916505 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916538 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916701 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.916760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916755 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.917011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.916841 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.918640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.918617 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.918852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.918836 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.972336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.972261 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:02.979636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:02.979596 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jfq\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq\") pod \"e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:03.018768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.018718 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:03.052761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.051608 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh"] Nov 02 08:17:03.056544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.056505 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.123371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.123332 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh"] Nov 02 08:17:03.123832 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.123809 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-09e3828d-openshift-default-dockercfg-27s59\"" Nov 02 08:17:03.226293 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.226203 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.226540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.226504 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.226727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.226708 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.226898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.226880 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.227058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.227041 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.227177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.227161 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.227333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.227318 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.227426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.227413 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.227584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.227569 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvzd\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.328977 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329025 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329047 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329085 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329217 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329255 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329316 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329343 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329413 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvzd\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329442 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329734 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.329936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329898 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.330042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.329959 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.330351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.330331 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.331716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.331696 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.332084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.332066 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.383856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.383827 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.384006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.383893 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvzd\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd\") pod \"e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.461894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.461858 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm"] Nov 02 08:17:03.463698 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:03.463670 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdc734b_e64e_49ed_b41b_15a7e9fdd029.slice/crio-c1702b2535ea5fa79cf453f3f3675d91a86da10a80ade695604a7577a90c181b WatchSource:0}: Error finding container c1702b2535ea5fa79cf453f3f3675d91a86da10a80ade695604a7577a90c181b: Status 404 returned error can't find the container with id c1702b2535ea5fa79cf453f3f3675d91a86da10a80ade695604a7577a90c181b Nov 02 08:17:03.465779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.465746 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:03.465859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.465824 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:03.465859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.465852 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:03.653208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.653169 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" event={"ID":"cfdc734b-e64e-49ed-b41b-15a7e9fdd029","Type":"ContainerStarted","Data":"fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa"} Nov 02 08:17:03.653367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.653232 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" event={"ID":"cfdc734b-e64e-49ed-b41b-15a7e9fdd029","Type":"ContainerStarted","Data":"c1702b2535ea5fa79cf453f3f3675d91a86da10a80ade695604a7577a90c181b"} Nov 02 08:17:03.675324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.675294 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:03.772237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:03.772179 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podStartSLOduration=1.772131143 podStartE2EDuration="1.772131143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:03.770473419 +0000 UTC m=+670.861093838" watchObservedRunningTime="2026-11-02 08:17:03.772131143 +0000 UTC m=+670.862751559" Nov 02 08:17:04.019757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.019633 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:04.024527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.024506 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:04.109618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.109588 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh"] Nov 02 08:17:04.111584 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:04.111522 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bffe6b4_659c_4111_9756_9c60980a9c4f.slice/crio-dcdc0576416f2282d7fe72b3128eb1d3945206ee70129cf79da26060e92a5cc1 WatchSource:0}: Error finding container dcdc0576416f2282d7fe72b3128eb1d3945206ee70129cf79da26060e92a5cc1: Status 404 returned error can't find the container with id dcdc0576416f2282d7fe72b3128eb1d3945206ee70129cf79da26060e92a5cc1 Nov 02 08:17:04.114496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.114461 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:04.114571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.114544 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:04.114610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.114582 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:04.660413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.660373 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" event={"ID":"6bffe6b4-659c-4111-9756-9c60980a9c4f","Type":"ContainerStarted","Data":"0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e"} Nov 02 08:17:04.660618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.660432 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" event={"ID":"6bffe6b4-659c-4111-9756-9c60980a9c4f","Type":"ContainerStarted","Data":"dcdc0576416f2282d7fe72b3128eb1d3945206ee70129cf79da26060e92a5cc1"} Nov 02 08:17:04.660724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.660705 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:04.661764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.661740 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:04.676030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.675992 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:04.789919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:04.789797 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podStartSLOduration=1.789778847 podStartE2EDuration="1.789778847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:04.787893659 +0000 UTC m=+671.878514078" watchObservedRunningTime="2026-11-02 08:17:04.789778847 +0000 UTC m=+671.880399269" Nov 02 08:17:05.680878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:05.680846 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:06.671604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:06.671573 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:06.672545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:06.672520 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:10.441217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.441175 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8"] Nov 02 08:17:10.445154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.445113 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.520509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.516770 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8"] Nov 02 08:17:10.567400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.567370 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d0628e-openshift-default-dockercfg-qvj7g\"" Nov 02 08:17:10.618908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.618874 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.618918 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619045 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619072 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619103 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619131 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619190 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619295 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6fr\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.619349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.619326 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720360 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720325 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6fr\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720371 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720407 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720430 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720470 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720487 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720506 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720529 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720588 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.720975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.720931 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.721057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.721028 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.721120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.721068 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.721320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.721301 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.721377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.721310 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.722827 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.722797 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.723048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.723029 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.775085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.775040 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:10.775372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:10.775348 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6fr\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:11.058895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.058805 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:11.480093 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.479720 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8"] Nov 02 08:17:11.483164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.480870 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:11.483164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.481017 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:11.483164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.481108 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:11.699899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.699859 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" event={"ID":"d0a85325-9fcb-4782-98bb-09b5e92fc1db","Type":"ContainerStarted","Data":"3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261"} Nov 02 08:17:11.700077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.699921 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" event={"ID":"d0a85325-9fcb-4782-98bb-09b5e92fc1db","Type":"ContainerStarted","Data":"65490ec6838460683fbfc1ed91c3061dfe10dbfeb94f2a0cd1a553854d14b219"} Nov 02 08:17:11.810721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:11.810601 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podStartSLOduration=1.8105811809999999 podStartE2EDuration="1.810581181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:11.809043379 +0000 UTC m=+678.899663797" watchObservedRunningTime="2026-11-02 08:17:11.810581181 +0000 UTC m=+678.901201601" Nov 02 08:17:12.059217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.059179 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:12.064313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.064246 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:12.683932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.683903 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg"] Nov 02 08:17:12.687879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.687856 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.705570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.705538 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:12.706522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.706502 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:17:12.736545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.736509 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2e734c1c-openshift-default-dockercfg-drp9l\"" Nov 02 08:17:12.742704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742668 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.742898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742786 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.742898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742818 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.742898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742840 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gznx8\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.742898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742874 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.743107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742911 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.743107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742935 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.743107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.742966 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.743107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.743051 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.761425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.761388 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg"] Nov 02 08:17:12.844248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844247 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844268 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gznx8\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844300 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844323 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844340 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844374 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844447 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.844560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844530 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.845053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.844890 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.845118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.845092 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.845213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.845189 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.845304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.845277 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.846265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.846242 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.847716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.847692 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.848381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.847806 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.902958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.902109 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gznx8\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:12.903270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:12.903242 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token\") pod \"e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:13.002126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.002094 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:13.440406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.440375 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg"] Nov 02 08:17:13.441663 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:13.441623 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d06009c_3094_4803_aa06_c283ff522cc7.slice/crio-9a76ae44079f24a52cf346d68c15ab99cb169d3e6cb975901d4ee6720ff7a1b5 WatchSource:0}: Error finding container 9a76ae44079f24a52cf346d68c15ab99cb169d3e6cb975901d4ee6720ff7a1b5: Status 404 returned error can't find the container with id 9a76ae44079f24a52cf346d68c15ab99cb169d3e6cb975901d4ee6720ff7a1b5 Nov 02 08:17:13.444057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.444023 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:13.444217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.444084 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:13.444217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.444115 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:13.712599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.712559 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" event={"ID":"2d06009c-3094-4803-aa06-c283ff522cc7","Type":"ContainerStarted","Data":"b0a96b1053238b62f0c08a717f4746327098a098a3a010723481181568f86424"} Nov 02 08:17:13.713078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.712619 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" event={"ID":"2d06009c-3094-4803-aa06-c283ff522cc7","Type":"ContainerStarted","Data":"9a76ae44079f24a52cf346d68c15ab99cb169d3e6cb975901d4ee6720ff7a1b5"} Nov 02 08:17:13.825829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:13.825767 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podStartSLOduration=1.825752451 podStartE2EDuration="1.825752451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:13.823432937 +0000 UTC m=+680.914053356" watchObservedRunningTime="2026-11-02 08:17:13.825752451 +0000 UTC m=+680.916372869" Nov 02 08:17:14.002999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.002883 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:14.008608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.008581 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:14.602828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.602792 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5"] Nov 02 08:17:14.607327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.607305 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.658891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.658851 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-09e3828d\"" Nov 02 08:17:14.659200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.659177 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-09e3828d-dockercfg-5k76x\"" Nov 02 08:17:14.667749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.664871 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bst6\" (UniqueName: \"kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6\") pod \"payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.667749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.664997 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.686605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.686563 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5"] Nov 02 08:17:14.736762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.736728 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:14.738227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.738203 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:17:14.766248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.766209 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bst6\" (UniqueName: \"kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6\") pod \"payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.766626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.766598 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.767608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.767583 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.814390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.814353 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl"] Nov 02 08:17:14.818482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.818456 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.821075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.821049 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bst6\" (UniqueName: \"kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6\") pod \"payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.867657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867577 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.867657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867628 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.867891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867741 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.867891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.868021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867891 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.868021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867963 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.868021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.867990 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.868021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.868014 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.868265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.868058 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.875337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.875306 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e6ca80-openshift-default-dockercfg-kspcn\"" Nov 02 08:17:14.905650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.905616 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl"] Nov 02 08:17:14.923104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.923052 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:14.969716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.969680 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.969923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.969730 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.969923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.969857 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.969923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.969902 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.969971 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.969990 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970027 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970084 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970309 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.970419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970352 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.971542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970541 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.971542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.970582 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.973289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.973251 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.973907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.973866 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:14.973907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:14.973886 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:15.029703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.029668 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:15.030041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.030014 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:15.141488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.141391 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:15.302532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.302103 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5"] Nov 02 08:17:15.304534 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:15.304476 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3176c3ba_f417_4339_883e_6ad788347e38.slice/crio-522019a7710fbb5a9ec2bbec0f70cc6ab4344f6dc0e263934766c71fae7fd2cc WatchSource:0}: Error finding container 522019a7710fbb5a9ec2bbec0f70cc6ab4344f6dc0e263934766c71fae7fd2cc: Status 404 returned error can't find the container with id 522019a7710fbb5a9ec2bbec0f70cc6ab4344f6dc0e263934766c71fae7fd2cc Nov 02 08:17:15.517333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.517300 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl"] Nov 02 08:17:15.518335 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:15.518301 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40b64d2_794e_4276_83e7_a68f0526e5e8.slice/crio-14f1287c6bccfd0a164ff9be57adc3760ab45c4f52b8819153d7fe530fdb3268 WatchSource:0}: Error finding container 14f1287c6bccfd0a164ff9be57adc3760ab45c4f52b8819153d7fe530fdb3268: Status 404 returned error can't find the container with id 14f1287c6bccfd0a164ff9be57adc3760ab45c4f52b8819153d7fe530fdb3268 Nov 02 08:17:15.523503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.523467 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:15.523608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.523535 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:15.523608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.523586 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:15.744126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.744089 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" event={"ID":"3176c3ba-f417-4339-883e-6ad788347e38","Type":"ContainerStarted","Data":"522019a7710fbb5a9ec2bbec0f70cc6ab4344f6dc0e263934766c71fae7fd2cc"} Nov 02 08:17:15.746229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.746171 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" event={"ID":"e40b64d2-794e-4276-83e7-a68f0526e5e8","Type":"ContainerStarted","Data":"db700d276bcdbcd86b83a7695c66010528eb7490a3712843f3070af2fdce5636"} Nov 02 08:17:15.746405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.746253 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" event={"ID":"e40b64d2-794e-4276-83e7-a68f0526e5e8","Type":"ContainerStarted","Data":"14f1287c6bccfd0a164ff9be57adc3760ab45c4f52b8819153d7fe530fdb3268"} Nov 02 08:17:15.862665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:15.862567 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podStartSLOduration=1.862551794 podStartE2EDuration="1.862551794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:15.859747509 +0000 UTC m=+682.950367928" watchObservedRunningTime="2026-11-02 08:17:15.862551794 +0000 UTC m=+682.953172213" Nov 02 08:17:16.141775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:16.141696 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:16.146994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:16.146965 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:16.754693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:16.754657 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:16.755670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:16.755646 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:17:18.766747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:18.766702 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" event={"ID":"3176c3ba-f417-4339-883e-6ad788347e38","Type":"ContainerStarted","Data":"e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688"} Nov 02 08:17:18.767166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:18.766813 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:18.878523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:18.878467 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" podStartSLOduration=1.744418815 podStartE2EDuration="4.878452445s" totalImagesPullingTime="3.13403363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:18.875733971 +0000 UTC m=+685.966354391" watchObservedRunningTime="2026-11-02 08:17:18.878452445 +0000 UTC m=+685.969072865" Nov 02 08:17:24.926560 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:17:24.926514 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:17:24.990960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:24.990919 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr"] Nov 02 08:17:24.995999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:24.995920 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.047246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.047216 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr"] Nov 02 08:17:25.087733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.087693 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6szw\" (UniqueName: \"kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw\") pod \"payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.087935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.087853 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.092854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.090270 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d0628e\"" Nov 02 08:17:25.092854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.090339 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d0628e-dockercfg-vdlgn\"" Nov 02 08:17:25.189424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.189330 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.189595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.189456 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6szw\" (UniqueName: \"kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw\") pod \"payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.190171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.190127 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.245670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.245641 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6szw\" (UniqueName: \"kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw\") pod \"payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.318202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.314311 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.407860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.407822 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:17:25.717341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.717301 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr"] Nov 02 08:17:25.722180 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:25.719483 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da911be_0240_403d_ab93_ff1edab16626.slice/crio-61581ed810cf5634a5112f84287f6515da4ded9f39f00776aa99c36ca0bb3dff WatchSource:0}: Error finding container 61581ed810cf5634a5112f84287f6515da4ded9f39f00776aa99c36ca0bb3dff: Status 404 returned error can't find the container with id 61581ed810cf5634a5112f84287f6515da4ded9f39f00776aa99c36ca0bb3dff Nov 02 08:17:25.816488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.816449 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" event={"ID":"8da911be-0240-403d-ab93-ff1edab16626","Type":"ContainerStarted","Data":"ab0e54afc03e58a652919c6581702dea42c7b1d12dc4bffef821c4f91f8d137b"} Nov 02 08:17:25.816642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.816513 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" event={"ID":"8da911be-0240-403d-ab93-ff1edab16626","Type":"ContainerStarted","Data":"61581ed810cf5634a5112f84287f6515da4ded9f39f00776aa99c36ca0bb3dff"} Nov 02 08:17:25.816642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.816558 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:25.925135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:25.925078 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" podStartSLOduration=1.925063465 podStartE2EDuration="1.925063465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:25.922071564 +0000 UTC m=+693.012691960" watchObservedRunningTime="2026-11-02 08:17:25.925063465 +0000 UTC m=+693.015683883" Nov 02 08:17:29.777576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:29.777546 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:17:30.009176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.006259 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d"] Nov 02 08:17:30.058223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.055311 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d"] Nov 02 08:17:30.058223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.055442 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.114165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.112058 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2e734c1c-dockercfg-zz8mc\"" Nov 02 08:17:30.114165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.112175 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2e734c1c\"" Nov 02 08:17:30.142686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.142637 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.142884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.142801 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pkd\" (UniqueName: \"kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd\") pod \"payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.244779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.244736 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pkd\" (UniqueName: \"kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd\") pod \"payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.244951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.244845 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.245875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.245846 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.325326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.325245 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pkd\" (UniqueName: \"kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd\") pod \"payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.373334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.373298 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:30.842432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.842400 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d"] Nov 02 08:17:30.844332 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:30.844301 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3b7cc4_0147_4311_b394_44a63a6aa970.slice/crio-57e0d19e9e2354144038338779dd8350706cbbf0ed399c66b8e45df145946277 WatchSource:0}: Error finding container 57e0d19e9e2354144038338779dd8350706cbbf0ed399c66b8e45df145946277: Status 404 returned error can't find the container with id 57e0d19e9e2354144038338779dd8350706cbbf0ed399c66b8e45df145946277 Nov 02 08:17:30.864735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:30.864709 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" event={"ID":"8f3b7cc4-0147-4311-b394-44a63a6aa970","Type":"ContainerStarted","Data":"57e0d19e9e2354144038338779dd8350706cbbf0ed399c66b8e45df145946277"} Nov 02 08:17:31.873432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:31.873396 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" event={"ID":"8f3b7cc4-0147-4311-b394-44a63a6aa970","Type":"ContainerStarted","Data":"6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1"} Nov 02 08:17:31.873831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:31.873496 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:31.985068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:31.985010 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" podStartSLOduration=2.984997036 podStartE2EDuration="2.984997036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:31.982297082 +0000 UTC m=+699.072917502" watchObservedRunningTime="2026-11-02 08:17:31.984997036 +0000 UTC m=+699.075617454" Nov 02 08:17:32.405359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.405258 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2"] Nov 02 08:17:32.409599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.409573 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.459894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.459814 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-16457c0e\"" Nov 02 08:17:32.463561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.463523 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2"] Nov 02 08:17:32.470184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.469413 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh95t\" (UniqueName: \"kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t\") pod \"payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.470184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.469687 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.510166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.505688 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-16457c0e-dockercfg-xrz5b\"" Nov 02 08:17:32.570925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.570880 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.571096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.570987 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh95t\" (UniqueName: \"kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t\") pod \"payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.571667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.571636 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.633495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.633461 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh95t\" (UniqueName: \"kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t\") pod \"payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:32.727111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:32.727068 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:33.112049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:33.111945 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2"] Nov 02 08:17:33.121245 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:33.120110 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c971b3_feda_4799_9c1d_ea133ca67d29.slice/crio-16544e3a0f1672bf8359a4ccf7045dbf8a71111aba0c2551fd469043c69687f5 WatchSource:0}: Error finding container 16544e3a0f1672bf8359a4ccf7045dbf8a71111aba0c2551fd469043c69687f5: Status 404 returned error can't find the container with id 16544e3a0f1672bf8359a4ccf7045dbf8a71111aba0c2551fd469043c69687f5 Nov 02 08:17:33.888805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:33.888762 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" event={"ID":"41c971b3-feda-4799-9c1d-ea133ca67d29","Type":"ContainerStarted","Data":"ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3"} Nov 02 08:17:33.888976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:33.888836 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" event={"ID":"41c971b3-feda-4799-9c1d-ea133ca67d29","Type":"ContainerStarted","Data":"16544e3a0f1672bf8359a4ccf7045dbf8a71111aba0c2551fd469043c69687f5"} Nov 02 08:17:33.889114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:33.889091 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:33.998928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:33.998861 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" podStartSLOduration=1.99884225 podStartE2EDuration="1.99884225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:33.994315101 +0000 UTC m=+701.084935548" watchObservedRunningTime="2026-11-02 08:17:33.99884225 +0000 UTC m=+701.089462671" Nov 02 08:17:34.578675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:34.578638 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:17:35.702345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.702287 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:17:35.703860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.702368 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:17:35.904812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.904721 5055 generic.go:356] "Generic (PLEG): container finished" podID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerID="0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e" exitCode=0 Nov 02 08:17:35.904812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.904793 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" event={"ID":"6bffe6b4-659c-4111-9756-9c60980a9c4f","Type":"ContainerDied","Data":"0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e"} Nov 02 08:17:35.905275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.905253 5055 scope.go:122] "RemoveContainer" containerID="0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e" Nov 02 08:17:35.906649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.906627 5055 generic.go:356] "Generic (PLEG): container finished" podID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerID="fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa" exitCode=0 Nov 02 08:17:35.906649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.906633 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:35.906801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.906695 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" event={"ID":"cfdc734b-e64e-49ed-b41b-15a7e9fdd029","Type":"ContainerDied","Data":"fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa"} Nov 02 08:17:35.906801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.906738 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:35.906801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.906788 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:35.907012 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.906997 5055 scope.go:122] "RemoveContainer" containerID="fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa" Nov 02 08:17:35.908343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.908317 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:35.908415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.908389 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:35.908459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:35.908436 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:36.672014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:36.671968 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:36.838069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:36.838026 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:17:36.935380 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:36.935268 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" event={"ID":"6bffe6b4-659c-4111-9756-9c60980a9c4f","Type":"ContainerStarted","Data":"26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e"} Nov 02 08:17:36.937185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:36.937159 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" event={"ID":"cfdc734b-e64e-49ed-b41b-15a7e9fdd029","Type":"ContainerStarted","Data":"2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d"} Nov 02 08:17:37.025984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:37.020707 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:37.025984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:37.020866 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 08:17:37.025984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:37.020913 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 08:17:37.676186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:37.676153 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:37.677355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:37.677334 5055 patch_prober.go:29] interesting pod/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 08:17:37.677428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:37.677381 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 08:17:38.022171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:38.021539 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 08:17:38.022171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:38.021603 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 08:17:38.676263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:38.676221 5055 patch_prober.go:29] interesting pod/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 08:17:38.676453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:38.676277 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 08:17:39.020611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:39.020568 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 08:17:39.020951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:39.020910 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 08:17:39.676452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:39.676417 5055 patch_prober.go:29] interesting pod/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 08:17:39.676881 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:39.676472 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 08:17:40.022164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:40.021515 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 08:17:40.022164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:40.021582 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 08:17:40.676029 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:40.675988 5055 patch_prober.go:29] interesting pod/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 08:17:40.676240 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:40.676043 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 08:17:41.022164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:41.021533 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 08:17:41.022164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:41.021597 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 08:17:41.675913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:41.675874 5055 patch_prober.go:29] interesting pod/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 08:17:41.676188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:41.676165 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 08:17:42.019745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.019706 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 08:17:42.019957 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.019770 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 08:17:42.627818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.627782 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh"] Nov 02 08:17:42.634381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.634357 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.680551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.680513 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:42.680914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.680886 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:42.681642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.681620 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:17:42.683216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.683200 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:17:42.729069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.729039 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:17:42.729264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.729167 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:17:42.729264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.729232 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:17:42.731739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.731719 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh"] Nov 02 08:17:42.757596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.757568 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.757751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.757603 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.757814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.757778 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpsk\" (UniqueName: \"kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.859156 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.859107 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpsk\" (UniqueName: \"kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.859381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.859265 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.859381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.859319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.859918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.859895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.861717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.861698 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.883043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.882975 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:17:42.916933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.916902 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpsk\" (UniqueName: \"kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk\") pod \"maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:42.948628 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:42.948597 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:43.038167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:43.034636 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:43.038167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:43.035282 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:43.038167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:43.036037 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:17:43.342531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:43.342454 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh"] Nov 02 08:17:43.346629 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:17:43.346586 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5cfa93_b0e8_433a_b438_409b753208d2.slice/crio-e6d204d007e8f607ce4d7bb101926c3ce1e0158f50959b82e804c4248178d248 WatchSource:0}: Error finding container e6d204d007e8f607ce4d7bb101926c3ce1e0158f50959b82e804c4248178d248: Status 404 returned error can't find the container with id e6d204d007e8f607ce4d7bb101926c3ce1e0158f50959b82e804c4248178d248 Nov 02 08:17:43.992656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:43.992621 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" event={"ID":"cd5cfa93-b0e8-433a-b438-409b753208d2","Type":"ContainerStarted","Data":"e6d204d007e8f607ce4d7bb101926c3ce1e0158f50959b82e804c4248178d248"} Nov 02 08:17:44.899278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:44.899250 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:17:48.020604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:48.020563 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" event={"ID":"cd5cfa93-b0e8-433a-b438-409b753208d2","Type":"ContainerStarted","Data":"2c464705c858af0337442c5ca43dfe1c58b0d83b68a1337b10d1668c9ce14b49"} Nov 02 08:17:48.126613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:48.126552 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" podStartSLOduration=2.107477293 podStartE2EDuration="6.126538239s" totalImagesPullingTime="4.019060946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:48.125849702 +0000 UTC m=+715.216470122" watchObservedRunningTime="2026-11-02 08:17:48.126538239 +0000 UTC m=+715.217158656" Nov 02 08:17:48.892572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:48.892533 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:17:49.028365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:49.028324 5055 generic.go:356] "Generic (PLEG): container finished" podID="cd5cfa93-b0e8-433a-b438-409b753208d2" containerID="2c464705c858af0337442c5ca43dfe1c58b0d83b68a1337b10d1668c9ce14b49" exitCode=0 Nov 02 08:17:49.028365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:49.028363 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" event={"ID":"cd5cfa93-b0e8-433a-b438-409b753208d2","Type":"ContainerDied","Data":"2c464705c858af0337442c5ca43dfe1c58b0d83b68a1337b10d1668c9ce14b49"} Nov 02 08:17:50.241162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.241126 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:50.245496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.245473 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca\") pod \"cd5cfa93-b0e8-433a-b438-409b753208d2\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " Nov 02 08:17:50.245608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.245530 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca\") pod \"cd5cfa93-b0e8-433a-b438-409b753208d2\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " Nov 02 08:17:50.245608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.245578 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk\" (UniqueName: \"kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk\") pod \"cd5cfa93-b0e8-433a-b438-409b753208d2\" (UID: \"cd5cfa93-b0e8-433a-b438-409b753208d2\") " Nov 02 08:17:50.245889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.245863 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca" pod "cd5cfa93-b0e8-433a-b438-409b753208d2" (UID: "cd5cfa93-b0e8-433a-b438-409b753208d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:17:50.247796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.247756 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca" pod "cd5cfa93-b0e8-433a-b438-409b753208d2" (UID: "cd5cfa93-b0e8-433a-b438-409b753208d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:17:50.247900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.247811 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk" pod "cd5cfa93-b0e8-433a-b438-409b753208d2" (UID: "cd5cfa93-b0e8-433a-b438-409b753208d2"). InnerVolumeSpecName "kube-api-access-hnpsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:17:50.347313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.347214 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:17:50.347313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.347245 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cfa93-b0e8-433a-b438-409b753208d2-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:17:50.347313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:50.347256 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpsk\" (UniqueName: \"kubernetes.io/projected/cd5cfa93-b0e8-433a-b438-409b753208d2-kube-api-access-hnpsk\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:17:51.041856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:51.041825 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" Nov 02 08:17:51.042035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:51.041865 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh" event={"ID":"cd5cfa93-b0e8-433a-b438-409b753208d2","Type":"ContainerDied","Data":"e6d204d007e8f607ce4d7bb101926c3ce1e0158f50959b82e804c4248178d248"} Nov 02 08:17:51.042035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:51.041915 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d204d007e8f607ce4d7bb101926c3ce1e0158f50959b82e804c4248178d248" Nov 02 08:17:54.923851 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:17:54.923802 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:17:57.509173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:57.508183 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:17:58.116218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:58.116174 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:45742->10.132.0.59:15021: read: connection reset by peer" start-of-body= Nov 02 08:17:58.116397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:58.116266 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:45742->10.132.0.59:15021: read: connection reset by peer" Nov 02 08:17:59.112799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:59.112761 5055 generic.go:356] "Generic (PLEG): container finished" podID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerID="3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261" exitCode=0 Nov 02 08:17:59.113231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:59.112828 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" event={"ID":"d0a85325-9fcb-4782-98bb-09b5e92fc1db","Type":"ContainerDied","Data":"3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261"} Nov 02 08:17:59.113231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:59.113221 5055 scope.go:122] "RemoveContainer" containerID="3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261" Nov 02 08:17:59.114601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:59.114575 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:59.114662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:59.114641 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:17:59.114696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:17:59.114670 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:00.147645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:00.147610 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" event={"ID":"d0a85325-9fcb-4782-98bb-09b5e92fc1db","Type":"ContainerStarted","Data":"1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617"} Nov 02 08:18:01.060010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:01.059976 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:18:01.060368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:01.060267 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 08:18:01.060505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:01.060488 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 08:18:02.059586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:02.059551 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 08:18:02.059991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:02.059608 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 08:18:03.059680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:03.059641 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 08:18:03.060112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:03.059720 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 08:18:04.059383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:04.059345 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 08:18:04.059586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:04.059410 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 08:18:05.059229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:05.059167 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 08:18:05.059735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:05.059268 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 08:18:06.063391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:06.063354 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:18:06.063803 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:06.063770 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:18:06.064467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:06.064448 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:18:06.322008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:06.321919 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5"] Nov 02 08:18:07.751173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.751124 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct"] Nov 02 08:18:07.751692 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.751673 5055 cpu_manager.go:414] "containerMap: removing container" podUID="cd5cfa93-b0e8-433a-b438-409b753208d2" containerName="revoke-keys" Nov 02 08:18:07.751692 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.751689 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5cfa93-b0e8-433a-b438-409b753208d2" containerName="revoke-keys" Nov 02 08:18:07.751781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.751768 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5cfa93-b0e8-433a-b438-409b753208d2" containerName="revoke-keys" Nov 02 08:18:07.754694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.754665 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.805368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.805283 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:18:07.805639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.805585 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:18:07.851657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.851612 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:18:07.851860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.851672 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:18:07.854073 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.854041 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct"] Nov 02 08:18:07.855555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.855533 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.855716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.855694 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2rtc\" (UniqueName: \"kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.855829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.855813 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.959666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.958926 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2rtc\" (UniqueName: \"kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.959666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.959034 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.959666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.959063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.977266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.964003 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:07.977266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.968425 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:18:07.977266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:07.975117 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:08.021869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.021836 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2rtc\" (UniqueName: \"kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc\") pod \"maas-api-revoke-keys-e2e-ait-2b62c067-8ckct\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:08.071220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.071183 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:08.208024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.203866 5055 generic.go:356] "Generic (PLEG): container finished" podID="2d06009c-3094-4803-aa06-c283ff522cc7" containerID="b0a96b1053238b62f0c08a717f4746327098a098a3a010723481181568f86424" exitCode=0 Nov 02 08:18:08.208024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.203937 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" event={"ID":"2d06009c-3094-4803-aa06-c283ff522cc7","Type":"ContainerDied","Data":"b0a96b1053238b62f0c08a717f4746327098a098a3a010723481181568f86424"} Nov 02 08:18:08.208024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.204368 5055 scope.go:122] "RemoveContainer" containerID="b0a96b1053238b62f0c08a717f4746327098a098a3a010723481181568f86424" Nov 02 08:18:08.208024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.205819 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:08.208024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.205885 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:08.208024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.205923 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:08.444741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:08.444537 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct"] Nov 02 08:18:08.445580 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:08.445445 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7113a5af_d3be_4996_b7e2_0a7e5205e149.slice/crio-e6776190c5cff77614667742c11a53be9ee624411e898fb412d59fbefa005aa2 WatchSource:0}: Error finding container e6776190c5cff77614667742c11a53be9ee624411e898fb412d59fbefa005aa2: Status 404 returned error can't find the container with id e6776190c5cff77614667742c11a53be9ee624411e898fb412d59fbefa005aa2 Nov 02 08:18:09.235419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:09.235355 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" event={"ID":"2d06009c-3094-4803-aa06-c283ff522cc7","Type":"ContainerStarted","Data":"760ea10b27142095bef8ad0ee2cdaedcbca4502b2e52d1cce8a69b3cfcb7b336"} Nov 02 08:18:09.237270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:09.237242 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" event={"ID":"7113a5af-d3be-4996-b7e2-0a7e5205e149","Type":"ContainerStarted","Data":"00df14d9cbd04e9a520f13e7ec3ad2c9d343c9191a2c6ea43fab7690d0184ab1"} Nov 02 08:18:09.237398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:09.237289 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" event={"ID":"7113a5af-d3be-4996-b7e2-0a7e5205e149","Type":"ContainerStarted","Data":"e6776190c5cff77614667742c11a53be9ee624411e898fb412d59fbefa005aa2"} Nov 02 08:18:09.464708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:09.464656 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" podStartSLOduration=2.464635091 podStartE2EDuration="2.464635091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:09.462965417 +0000 UTC m=+736.553585833" watchObservedRunningTime="2026-11-02 08:18:09.464635091 +0000 UTC m=+736.555255510" Nov 02 08:18:10.005529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:10.005491 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:18:10.014162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:10.013261 5055 patch_prober.go:29] interesting pod/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 08:18:10.014162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:10.013322 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 08:18:10.245611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:10.245580 5055 generic.go:356] "Generic (PLEG): container finished" podID="7113a5af-d3be-4996-b7e2-0a7e5205e149" containerID="00df14d9cbd04e9a520f13e7ec3ad2c9d343c9191a2c6ea43fab7690d0184ab1" exitCode=0 Nov 02 08:18:10.246462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:10.246436 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" event={"ID":"7113a5af-d3be-4996-b7e2-0a7e5205e149","Type":"ContainerDied","Data":"00df14d9cbd04e9a520f13e7ec3ad2c9d343c9191a2c6ea43fab7690d0184ab1"} Nov 02 08:18:11.005224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.005183 5055 patch_prober.go:29] interesting pod/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 08:18:11.005553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.005519 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 08:18:11.335670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.335553 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" podUID="3176c3ba-f417-4339-883e-6ad788347e38" containerName="payload-pre-processing" containerID="cri-o://e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688" gracePeriod=25 Nov 02 08:18:11.546186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.545930 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:11.648016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.644308 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca\") pod \"7113a5af-d3be-4996-b7e2-0a7e5205e149\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " Nov 02 08:18:11.648016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.644376 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca\") pod \"7113a5af-d3be-4996-b7e2-0a7e5205e149\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " Nov 02 08:18:11.648016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.644431 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc\" (UniqueName: \"kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc\") pod \"7113a5af-d3be-4996-b7e2-0a7e5205e149\" (UID: \"7113a5af-d3be-4996-b7e2-0a7e5205e149\") " Nov 02 08:18:11.648016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.645231 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca" pod "7113a5af-d3be-4996-b7e2-0a7e5205e149" (UID: "7113a5af-d3be-4996-b7e2-0a7e5205e149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:11.648016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.646930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca" pod "7113a5af-d3be-4996-b7e2-0a7e5205e149" (UID: "7113a5af-d3be-4996-b7e2-0a7e5205e149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:11.648016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.647897 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc" pod "7113a5af-d3be-4996-b7e2-0a7e5205e149" (UID: "7113a5af-d3be-4996-b7e2-0a7e5205e149"). InnerVolumeSpecName "kube-api-access-b2rtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:11.744765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.744734 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:18:11.745600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.745574 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:11.745600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.745598 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113a5af-d3be-4996-b7e2-0a7e5205e149-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:11.745815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.745608 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2rtc\" (UniqueName: \"kubernetes.io/projected/7113a5af-d3be-4996-b7e2-0a7e5205e149-kube-api-access-b2rtc\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:11.846180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.846028 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6\" (UniqueName: \"kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6\") pod \"3176c3ba-f417-4339-883e-6ad788347e38\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " Nov 02 08:18:11.846180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.846090 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume\") pod \"3176c3ba-f417-4339-883e-6ad788347e38\" (UID: \"3176c3ba-f417-4339-883e-6ad788347e38\") " Nov 02 08:18:11.846580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.846553 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume" pod "3176c3ba-f417-4339-883e-6ad788347e38" (UID: "3176c3ba-f417-4339-883e-6ad788347e38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:11.848455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.848433 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6" pod "3176c3ba-f417-4339-883e-6ad788347e38" (UID: "3176c3ba-f417-4339-883e-6ad788347e38"). InnerVolumeSpecName "kube-api-access-8bst6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:11.949312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.949277 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bst6\" (UniqueName: \"kubernetes.io/projected/3176c3ba-f417-4339-883e-6ad788347e38-kube-api-access-8bst6\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:11.949312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:11.949310 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3176c3ba-f417-4339-883e-6ad788347e38-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:12.007165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.006598 5055 patch_prober.go:29] interesting pod/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 08:18:12.007165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.006664 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 08:18:12.260763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.260658 5055 generic.go:356] "Generic (PLEG): container finished" podID="3176c3ba-f417-4339-883e-6ad788347e38" containerID="e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688" exitCode=0 Nov 02 08:18:12.260763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.260702 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" Nov 02 08:18:12.260763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.260727 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" event={"ID":"3176c3ba-f417-4339-883e-6ad788347e38","Type":"ContainerDied","Data":"e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688"} Nov 02 08:18:12.261066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.260804 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5" event={"ID":"3176c3ba-f417-4339-883e-6ad788347e38","Type":"ContainerDied","Data":"522019a7710fbb5a9ec2bbec0f70cc6ab4344f6dc0e263934766c71fae7fd2cc"} Nov 02 08:18:12.261066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.260833 5055 scope.go:122] "RemoveContainer" containerID="e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688" Nov 02 08:18:12.263207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.263187 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" Nov 02 08:18:12.263321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.263251 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct" event={"ID":"7113a5af-d3be-4996-b7e2-0a7e5205e149","Type":"ContainerDied","Data":"e6776190c5cff77614667742c11a53be9ee624411e898fb412d59fbefa005aa2"} Nov 02 08:18:12.263321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.263289 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6776190c5cff77614667742c11a53be9ee624411e898fb412d59fbefa005aa2" Nov 02 08:18:12.273542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.273521 5055 scope.go:122] "RemoveContainer" containerID="e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688" Nov 02 08:18:12.273878 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:12.273859 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688\": container with ID starting with e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688 not found: ID does not exist" containerID="e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688" Nov 02 08:18:12.273938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.273889 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688"} err="failed to get container status \"e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688\": rpc error: code = NotFound desc = could not find container \"e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688\": container with ID starting with e4a773546427e19e812d53e3ed3ce4279566c5dba2edc227eb10715b40fb6688 not found: ID does not exist" Nov 02 08:18:12.439867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.439794 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5"] Nov 02 08:18:12.442861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:12.442834 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3828d-7d6cc9fd47-fvkw5"] Nov 02 08:18:13.005025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.004969 5055 patch_prober.go:29] interesting pod/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 08:18:13.005358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.005315 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.126296 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3176c3ba-f417-4339-883e-6ad788347e38" path="/var/lib/kubelet/pods/3176c3ba-f417-4339-883e-6ad788347e38/volumes" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.126741 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6"] Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.127468 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3176c3ba-f417-4339-883e-6ad788347e38" containerName="payload-pre-processing" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.127488 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3176c3ba-f417-4339-883e-6ad788347e38" containerName="payload-pre-processing" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.127566 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7113a5af-d3be-4996-b7e2-0a7e5205e149" containerName="revoke-keys" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.127576 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7113a5af-d3be-4996-b7e2-0a7e5205e149" containerName="revoke-keys" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.127688 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7113a5af-d3be-4996-b7e2-0a7e5205e149" containerName="revoke-keys" Nov 02 08:18:13.129195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.127701 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3176c3ba-f417-4339-883e-6ad788347e38" containerName="payload-pre-processing" Nov 02 08:18:13.141561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.141535 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.176008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.175942 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lffp\" (UniqueName: \"kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.176298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.176208 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.176599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.176567 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.179812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.179786 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6"] Nov 02 08:18:13.203571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.203538 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:18:13.229231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.208461 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:18:13.237792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.208563 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:18:13.238062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.224330 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:18:13.278232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.278102 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.278463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.278263 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lffp\" (UniqueName: \"kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.278463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.278322 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.279099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.279070 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.339667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.339626 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.347168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.345920 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lffp\" (UniqueName: \"kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp\") pod \"maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.456172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.455965 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:13.945178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:13.928830 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6"] Nov 02 08:18:13.964234 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:13.964195 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57761ea8_0cca_4009_be2a_a76637347f22.slice/crio-b495a120dded01a3f0e82d612b9731cceb16e775cc6c8303ed91d9dce434368e WatchSource:0}: Error finding container b495a120dded01a3f0e82d612b9731cceb16e775cc6c8303ed91d9dce434368e: Status 404 returned error can't find the container with id b495a120dded01a3f0e82d612b9731cceb16e775cc6c8303ed91d9dce434368e Nov 02 08:18:14.009179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:14.006594 5055 patch_prober.go:29] interesting pod/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 08:18:14.009179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:14.006661 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 08:18:14.300167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:14.294215 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" event={"ID":"57761ea8-0cca-4009-be2a-a76637347f22","Type":"ContainerStarted","Data":"b495a120dded01a3f0e82d612b9731cceb16e775cc6c8303ed91d9dce434368e"} Nov 02 08:18:14.736997 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:14.736953 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:18:15.008467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.008435 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:18:15.304370 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.304281 5055 generic.go:356] "Generic (PLEG): container finished" podID="57761ea8-0cca-4009-be2a-a76637347f22" containerID="87c47daf229be605b04327821baadee69a6db47c35bfd74ab1736a5d6ae193b9" exitCode=0 Nov 02 08:18:15.304506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.304374 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" event={"ID":"57761ea8-0cca-4009-be2a-a76637347f22","Type":"ContainerDied","Data":"87c47daf229be605b04327821baadee69a6db47c35bfd74ab1736a5d6ae193b9"} Nov 02 08:18:15.305634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.305610 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:18:15.723778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.723737 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh"] Nov 02 08:18:15.726836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.726803 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09e3828d-jqgsh"] Nov 02 08:18:15.768113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.768069 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh"] Nov 02 08:18:15.768936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:15.768903 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" containerID="cri-o://26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e" gracePeriod=30 Nov 02 08:18:16.461405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.461364 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:16.488146 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.488104 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca\") pod \"57761ea8-0cca-4009-be2a-a76637347f22\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " Nov 02 08:18:16.488309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.488174 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca\") pod \"57761ea8-0cca-4009-be2a-a76637347f22\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " Nov 02 08:18:16.488611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.488583 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca" pod "57761ea8-0cca-4009-be2a-a76637347f22" (UID: "57761ea8-0cca-4009-be2a-a76637347f22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:16.490248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.490218 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca" pod "57761ea8-0cca-4009-be2a-a76637347f22" (UID: "57761ea8-0cca-4009-be2a-a76637347f22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:16.589224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.589084 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp\" (UniqueName: \"kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp\") pod \"57761ea8-0cca-4009-be2a-a76637347f22\" (UID: \"57761ea8-0cca-4009-be2a-a76637347f22\") " Nov 02 08:18:16.589766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.589510 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:16.589766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.589535 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57761ea8-0cca-4009-be2a-a76637347f22-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:16.592231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.592186 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp" pod "57761ea8-0cca-4009-be2a-a76637347f22" (UID: "57761ea8-0cca-4009-be2a-a76637347f22"). InnerVolumeSpecName "kube-api-access-5lffp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:16.690554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:16.690516 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lffp\" (UniqueName: \"kubernetes.io/projected/57761ea8-0cca-4009-be2a-a76637347f22-kube-api-access-5lffp\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:17.051072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.051031 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx"] Nov 02 08:18:17.051610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.051595 5055 cpu_manager.go:414] "containerMap: removing container" podUID="57761ea8-0cca-4009-be2a-a76637347f22" containerName="revoke-keys" Nov 02 08:18:17.051672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.051611 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57761ea8-0cca-4009-be2a-a76637347f22" containerName="revoke-keys" Nov 02 08:18:17.051721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.051700 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="57761ea8-0cca-4009-be2a-a76637347f22" containerName="revoke-keys" Nov 02 08:18:17.056812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.056790 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102269 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vfq\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102350 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102393 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102419 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102460 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102489 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102528 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102567 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.105191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.102585 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.114082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.113771 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-10d2fb-openshift-default-dockercfg-hh68x\"" Nov 02 08:18:17.124432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.124396 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5cfa93-b0e8-433a-b438-409b753208d2" path="/var/lib/kubelet/pods/cd5cfa93-b0e8-433a-b438-409b753208d2/volumes" Nov 02 08:18:17.155739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.155705 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx"] Nov 02 08:18:17.213888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.213846 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.213899 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.213960 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.214127 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.214248 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.214309 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.214338 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.214422 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vfq\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.214644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.214474 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.215298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.215275 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.216003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.215625 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.216239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.215954 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.217702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.217032 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.217702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.217486 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.225184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.220730 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.225817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.225772 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.270881 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.270846 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.271398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.271368 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vfq\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.322474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.322376 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" event={"ID":"57761ea8-0cca-4009-be2a-a76637347f22","Type":"ContainerDied","Data":"b495a120dded01a3f0e82d612b9731cceb16e775cc6c8303ed91d9dce434368e"} Nov 02 08:18:17.322474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.322438 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b495a120dded01a3f0e82d612b9731cceb16e775cc6c8303ed91d9dce434368e" Nov 02 08:18:17.322698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.322512 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6" Nov 02 08:18:17.371260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.371220 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:17.796783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.796754 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx"] Nov 02 08:18:17.798416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.798293 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:18:17.798416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.798352 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:18:17.798649 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:17.798555 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c2b98a_18f5_4b2e_a54a_ce702a109380.slice/crio-b71ec2f619b0dcf4ae1ec5371789081e5395dbb0ff8ff1d6e990f68ad2610604 WatchSource:0}: Error finding container b71ec2f619b0dcf4ae1ec5371789081e5395dbb0ff8ff1d6e990f68ad2610604: Status 404 returned error can't find the container with id b71ec2f619b0dcf4ae1ec5371789081e5395dbb0ff8ff1d6e990f68ad2610604 Nov 02 08:18:17.801206 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.801171 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:17.801302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.801231 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:17.801302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:17.801262 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:18.333035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.332991 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" event={"ID":"88c2b98a-18f5-4b2e-a54a-ce702a109380","Type":"ContainerStarted","Data":"dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb"} Nov 02 08:18:18.333660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.333068 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" event={"ID":"88c2b98a-18f5-4b2e-a54a-ce702a109380","Type":"ContainerStarted","Data":"b71ec2f619b0dcf4ae1ec5371789081e5395dbb0ff8ff1d6e990f68ad2610604"} Nov 02 08:18:18.334981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.334952 5055 generic.go:356] "Generic (PLEG): container finished" podID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerID="db700d276bcdbcd86b83a7695c66010528eb7490a3712843f3070af2fdce5636" exitCode=0 Nov 02 08:18:18.335129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.335035 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" event={"ID":"e40b64d2-794e-4276-83e7-a68f0526e5e8","Type":"ContainerDied","Data":"db700d276bcdbcd86b83a7695c66010528eb7490a3712843f3070af2fdce5636"} Nov 02 08:18:18.335590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.335572 5055 scope.go:122] "RemoveContainer" containerID="db700d276bcdbcd86b83a7695c66010528eb7490a3712843f3070af2fdce5636" Nov 02 08:18:18.337250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.337211 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:18.337363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.337308 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:18.337363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.337355 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:18.371431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.371393 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:18.377792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.377762 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:18.465363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:18.465309 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podStartSLOduration=1.465292759 podStartE2EDuration="1.465292759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:18.461840763 +0000 UTC m=+745.552461181" watchObservedRunningTime="2026-11-02 08:18:18.465292759 +0000 UTC m=+745.555913174" Nov 02 08:18:19.343333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:19.343294 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" event={"ID":"e40b64d2-794e-4276-83e7-a68f0526e5e8","Type":"ContainerStarted","Data":"95b3c0922405fb4e8e2dad4027be0fb5a03b5ed147e3b8fabe405975fe34a17e"} Nov 02 08:18:19.343915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:19.343545 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:19.344542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:19.344522 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:18:20.142568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:20.142534 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:18:20.142767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:20.142746 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 08:18:20.142821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:20.142804 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 08:18:21.115914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.115882 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:18:21.142103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.142067 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 08:18:21.142286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.142123 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 08:18:21.169509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169479 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169539 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169567 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169600 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169635 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169766 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169786 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.169842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169808 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:21.169842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169815 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.170046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169903 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:21.170046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169910 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert\") pod \"6bffe6b4-659c-4111-9756-9c60980a9c4f\" (UID: \"6bffe6b4-659c-4111-9756-9c60980a9c4f\") " Nov 02 08:18:21.170196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.169924 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:21.170196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170039 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:21.170327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170231 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:21.170482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170458 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.170482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170487 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bffe6b4-659c-4111-9756-9c60980a9c4f-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.170617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170502 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.170617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170514 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.170617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.170528 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.172009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.171985 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:21.172118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.172026 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:21.172269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.172115 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "kube-api-access-2nvzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:21.172269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.172205 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo" pod "6bffe6b4-659c-4111-9756-9c60980a9c4f" (UID: "6bffe6b4-659c-4111-9756-9c60980a9c4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:18:21.272246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.272204 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nvzd\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-kube-api-access-2nvzd\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.272246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.272239 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.272488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.272263 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.272488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.272278 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bffe6b4-659c-4111-9756-9c60980a9c4f-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:21.357666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.357575 5055 generic.go:356] "Generic (PLEG): container finished" podID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerID="26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e" exitCode=0 Nov 02 08:18:21.357666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.357633 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" Nov 02 08:18:21.357889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.357658 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" event={"ID":"6bffe6b4-659c-4111-9756-9c60980a9c4f","Type":"ContainerDied","Data":"26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e"} Nov 02 08:18:21.357889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.357725 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh" event={"ID":"6bffe6b4-659c-4111-9756-9c60980a9c4f","Type":"ContainerDied","Data":"dcdc0576416f2282d7fe72b3128eb1d3945206ee70129cf79da26060e92a5cc1"} Nov 02 08:18:21.357889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.357749 5055 scope.go:122] "RemoveContainer" containerID="26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e" Nov 02 08:18:21.368815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.368790 5055 scope.go:122] "RemoveContainer" containerID="0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e" Nov 02 08:18:21.406384 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.406356 5055 scope.go:122] "RemoveContainer" containerID="26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e" Nov 02 08:18:21.407219 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:21.407131 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e\": container with ID starting with 26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e not found: ID does not exist" containerID="26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e" Nov 02 08:18:21.407341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.407231 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e"} err="failed to get container status \"26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e\": rpc error: code = NotFound desc = could not find container \"26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e\": container with ID starting with 26eac0d67e65259fb7ce7b87706ce76d5f848795ddc17128a13cd1775390412e not found: ID does not exist" Nov 02 08:18:21.407341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.407259 5055 scope.go:122] "RemoveContainer" containerID="0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e" Nov 02 08:18:21.407687 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:21.407652 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e\": container with ID starting with 0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e not found: ID does not exist" containerID="0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e" Nov 02 08:18:21.407762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.407699 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e"} err="failed to get container status \"0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e\": rpc error: code = NotFound desc = could not find container \"0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e\": container with ID starting with 0a5b37c7184762432ed59df5b3e4ca201822315f5ae68c431e0ded5908eb093e not found: ID does not exist" Nov 02 08:18:21.512362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.512317 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh"] Nov 02 08:18:21.517616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:21.517578 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-09e3828d-openshift-default-54b654cff9-79hnh"] Nov 02 08:18:22.142807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:22.142765 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 08:18:22.143229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:22.142829 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 08:18:23.122998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:23.122961 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" path="/var/lib/kubelet/pods/6bffe6b4-659c-4111-9756-9c60980a9c4f/volumes" Nov 02 08:18:23.142076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:23.142033 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 08:18:23.142289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:23.142107 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 08:18:24.141953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:24.141918 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 08:18:24.142391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:24.141982 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 08:18:24.919628 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:24.919584 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:18:25.145643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:25.145612 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:18:25.146222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:25.145910 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:18:25.146559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:25.146535 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:18:27.403638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:27.403598 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:18:31.347407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.347374 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs"] Nov 02 08:18:31.348185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.348164 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" Nov 02 08:18:31.348250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.348188 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" Nov 02 08:18:31.348297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.348256 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" Nov 02 08:18:31.348297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.348266 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" Nov 02 08:18:31.348434 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.348421 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" Nov 02 08:18:31.348483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.348442 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bffe6b4-659c-4111-9756-9c60980a9c4f" containerName="istio-proxy" Nov 02 08:18:31.352621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.352600 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.408316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.408279 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs"] Nov 02 08:18:31.409112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.409084 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.409362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.409343 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.409563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.409539 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.409683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.409657 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.409753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.409718 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.409841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.409819 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjscv\" (UniqueName: \"kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.450351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.450298 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"default-dockercfg-g4f7x\"" Nov 02 08:18:31.450558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.450418 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"simulator-w1-d0628e-kserve-self-signed-certs\"" Nov 02 08:18:31.450624 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.450566 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"kube-root-ca.crt\"" Nov 02 08:18:31.450624 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.450615 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"openshift-service-ca.crt\"" Nov 02 08:18:31.510981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.510945 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.510990 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511030 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjscv\" (UniqueName: \"kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511067 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511097 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511174 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511672 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511732 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.511830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.511767 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.513561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.513531 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.513817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.513799 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.564135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.564102 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjscv\" (UniqueName: \"kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv\") pod \"simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:31.664246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:31.664163 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:32.032838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.032810 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs"] Nov 02 08:18:32.034306 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:32.034269 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95370416_e4b9_4d7f_afb2_f052a9a30690.slice/crio-6b0c72455cfdf29bfd5dfa45933cfcd74cf563ed9445a19f3c5ebeff0fd82eab WatchSource:0}: Error finding container 6b0c72455cfdf29bfd5dfa45933cfcd74cf563ed9445a19f3c5ebeff0fd82eab: Status 404 returned error can't find the container with id 6b0c72455cfdf29bfd5dfa45933cfcd74cf563ed9445a19f3c5ebeff0fd82eab Nov 02 08:18:32.436576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.436456 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" event={"ID":"95370416-e4b9-4d7f-afb2-f052a9a30690","Type":"ContainerStarted","Data":"e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b"} Nov 02 08:18:32.436576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.436522 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" event={"ID":"95370416-e4b9-4d7f-afb2-f052a9a30690","Type":"ContainerStarted","Data":"6b0c72455cfdf29bfd5dfa45933cfcd74cf563ed9445a19f3c5ebeff0fd82eab"} Nov 02 08:18:32.546134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.545836 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podStartSLOduration=1.5458204169999998 podStartE2EDuration="1.545820417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:32.543478263 +0000 UTC m=+759.634098683" watchObservedRunningTime="2026-11-02 08:18:32.545820417 +0000 UTC m=+759.636440836" Nov 02 08:18:32.665873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.664357 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:32.897049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.897018 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc"] Nov 02 08:18:32.900216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.900197 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:32.924252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.924219 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:32.924412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.924287 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:32.924412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.924397 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:32.953514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.953475 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:18:32.998968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.998929 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:18:32.998968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.998937 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:18:32.999308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:32.998995 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:18:33.001631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.001609 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc"] Nov 02 08:18:33.025886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.025855 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.026080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.025949 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.026080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.026002 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.026805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.026780 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.028607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.028578 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.080342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.080303 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6\") pod \"maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.213442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.213408 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:33.586348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:33.586324 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc"] Nov 02 08:18:33.588878 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:33.588844 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903750b3_bdba_4e51_9066_82fb40d7d87e.slice/crio-8e5c584c62c925849e58fdcbbdd3b6944c9391c0c13b2d9ca1d730c240df942b WatchSource:0}: Error finding container 8e5c584c62c925849e58fdcbbdd3b6944c9391c0c13b2d9ca1d730c240df942b: Status 404 returned error can't find the container with id 8e5c584c62c925849e58fdcbbdd3b6944c9391c0c13b2d9ca1d730c240df942b Nov 02 08:18:34.451472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:34.451442 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" event={"ID":"903750b3-bdba-4e51-9066-82fb40d7d87e","Type":"ContainerStarted","Data":"1726067eadc0b1837a2b34729dd41f3367eb0948e525d0862edccd415eb8fad1"} Nov 02 08:18:34.451631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:34.451504 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" event={"ID":"903750b3-bdba-4e51-9066-82fb40d7d87e","Type":"ContainerStarted","Data":"8e5c584c62c925849e58fdcbbdd3b6944c9391c0c13b2d9ca1d730c240df942b"} Nov 02 08:18:35.458579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.458538 5055 generic.go:356] "Generic (PLEG): container finished" podID="903750b3-bdba-4e51-9066-82fb40d7d87e" containerID="1726067eadc0b1837a2b34729dd41f3367eb0948e525d0862edccd415eb8fad1" exitCode=0 Nov 02 08:18:35.459046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.458628 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" event={"ID":"903750b3-bdba-4e51-9066-82fb40d7d87e","Type":"ContainerDied","Data":"1726067eadc0b1837a2b34729dd41f3367eb0948e525d0862edccd415eb8fad1"} Nov 02 08:18:35.631251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.631205 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:35.654239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.654204 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca\") pod \"903750b3-bdba-4e51-9066-82fb40d7d87e\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " Nov 02 08:18:35.654416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.654258 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca\") pod \"903750b3-bdba-4e51-9066-82fb40d7d87e\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " Nov 02 08:18:35.654416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.654379 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6\") pod \"903750b3-bdba-4e51-9066-82fb40d7d87e\" (UID: \"903750b3-bdba-4e51-9066-82fb40d7d87e\") " Nov 02 08:18:35.654625 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.654598 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca" pod "903750b3-bdba-4e51-9066-82fb40d7d87e" (UID: "903750b3-bdba-4e51-9066-82fb40d7d87e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:35.654733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.654717 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:35.656433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.656409 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca" pod "903750b3-bdba-4e51-9066-82fb40d7d87e" (UID: "903750b3-bdba-4e51-9066-82fb40d7d87e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:35.656496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.656442 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6" pod "903750b3-bdba-4e51-9066-82fb40d7d87e" (UID: "903750b3-bdba-4e51-9066-82fb40d7d87e"). InnerVolumeSpecName "kube-api-access-zpcl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:35.755836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.755744 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903750b3-bdba-4e51-9066-82fb40d7d87e-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:35.755836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:35.755773 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/903750b3-bdba-4e51-9066-82fb40d7d87e-kube-api-access-zpcl6\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:36.465642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:36.465601 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" event={"ID":"903750b3-bdba-4e51-9066-82fb40d7d87e","Type":"ContainerDied","Data":"8e5c584c62c925849e58fdcbbdd3b6944c9391c0c13b2d9ca1d730c240df942b"} Nov 02 08:18:36.465642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:36.465642 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc" Nov 02 08:18:36.466107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:36.465663 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e5c584c62c925849e58fdcbbdd3b6944c9391c0c13b2d9ca1d730c240df942b" Nov 02 08:18:37.627516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:37.627461 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct"] Nov 02 08:18:37.668400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:37.668352 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b62c067-8ckct"] Nov 02 08:18:38.193650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:38.193608 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm"] Nov 02 08:18:38.193889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:38.193859 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" containerID="cri-o://2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d" gracePeriod=30 Nov 02 08:18:38.776052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:38.776010 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6"] Nov 02 08:18:38.778398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:38.778373 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53cc8e62-hj5r6"] Nov 02 08:18:39.092091 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.092006 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57761ea8-0cca-4009-be2a-a76637347f22" path="/var/lib/kubelet/pods/57761ea8-0cca-4009-be2a-a76637347f22/volumes" Nov 02 08:18:39.092584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.092561 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7113a5af-d3be-4996-b7e2-0a7e5205e149" path="/var/lib/kubelet/pods/7113a5af-d3be-4996-b7e2-0a7e5205e149/volumes" Nov 02 08:18:39.108614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.108584 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m"] Nov 02 08:18:39.109118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.109100 5055 cpu_manager.go:414] "containerMap: removing container" podUID="903750b3-bdba-4e51-9066-82fb40d7d87e" containerName="revoke-keys" Nov 02 08:18:39.109250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.109120 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903750b3-bdba-4e51-9066-82fb40d7d87e" containerName="revoke-keys" Nov 02 08:18:39.109334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.109319 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="903750b3-bdba-4e51-9066-82fb40d7d87e" containerName="revoke-keys" Nov 02 08:18:39.111958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.111938 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.161593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.161563 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:18:39.161716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.161656 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:18:39.204377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.204343 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6d4m\" (UniqueName: \"kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.204377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.204381 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.204584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.204524 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.207676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.207646 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:18:39.207676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.207656 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:18:39.210261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.210239 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m"] Nov 02 08:18:39.305117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.305082 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6d4m\" (UniqueName: \"kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.305298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.305122 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.305298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.305228 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.305864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.305844 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.307687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.307665 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.358910 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.358830 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6d4m\" (UniqueName: \"kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m\") pod \"maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.423936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.423900 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:39.898961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:39.898933 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m"] Nov 02 08:18:39.900301 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:39.900271 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1de7ff_0a18_4ff0_8b7d_c2f5e006fb19.slice/crio-015fd80c2cfbea51a812227c00edc7c43b8c36102f182393e87914b764f3c9f8 WatchSource:0}: Error finding container 015fd80c2cfbea51a812227c00edc7c43b8c36102f182393e87914b764f3c9f8: Status 404 returned error can't find the container with id 015fd80c2cfbea51a812227c00edc7c43b8c36102f182393e87914b764f3c9f8 Nov 02 08:18:40.500121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:40.500082 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" event={"ID":"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19","Type":"ContainerStarted","Data":"00ef773284e609a59a5f8ab6b84d4f49b70ff2454a27ba2455fa81d83594f4e7"} Nov 02 08:18:40.500322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:40.500129 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" event={"ID":"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19","Type":"ContainerStarted","Data":"015fd80c2cfbea51a812227c00edc7c43b8c36102f182393e87914b764f3c9f8"} Nov 02 08:18:40.618327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:40.618270 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" podStartSLOduration=1.618256685 podStartE2EDuration="1.618256685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:40.616244303 +0000 UTC m=+767.706864719" watchObservedRunningTime="2026-11-02 08:18:40.618256685 +0000 UTC m=+767.708877105" Nov 02 08:18:41.507068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:41.507035 5055 generic.go:356] "Generic (PLEG): container finished" podID="4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" containerID="00ef773284e609a59a5f8ab6b84d4f49b70ff2454a27ba2455fa81d83594f4e7" exitCode=0 Nov 02 08:18:41.507483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:41.507126 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" event={"ID":"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19","Type":"ContainerDied","Data":"00ef773284e609a59a5f8ab6b84d4f49b70ff2454a27ba2455fa81d83594f4e7"} Nov 02 08:18:41.664319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:41.664270 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:41.677892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:41.677859 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:42.525242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.525211 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:18:42.665850 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.665808 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:42.752069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.752039 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca\") pod \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " Nov 02 08:18:42.752283 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.752085 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m\" (UniqueName: \"kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m\") pod \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " Nov 02 08:18:42.752283 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.752117 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca\") pod \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\" (UID: \"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19\") " Nov 02 08:18:42.752518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.752491 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca" pod "4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" (UID: "4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:42.754313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.754280 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca" pod "4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" (UID: "4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:42.754313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.754299 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m" pod "4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" (UID: "4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19"). InnerVolumeSpecName "kube-api-access-l6d4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:42.853695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.853605 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:42.853695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.853640 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6d4m\" (UniqueName: \"kubernetes.io/projected/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-kube-api-access-l6d4m\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:42.853695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:42.853653 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.037475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.037431 5055 patch_prober.go:29] interesting pod/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:18:43.037653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.037516 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:18:43.459313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.459239 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:18:43.522944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.522899 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" Nov 02 08:18:43.522944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.522912 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m" event={"ID":"4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19","Type":"ContainerDied","Data":"015fd80c2cfbea51a812227c00edc7c43b8c36102f182393e87914b764f3c9f8"} Nov 02 08:18:43.523242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.522979 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015fd80c2cfbea51a812227c00edc7c43b8c36102f182393e87914b764f3c9f8" Nov 02 08:18:43.525106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.525070 5055 generic.go:356] "Generic (PLEG): container finished" podID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerID="2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d" exitCode=0 Nov 02 08:18:43.525285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.525119 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" Nov 02 08:18:43.525285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.525256 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" event={"ID":"cfdc734b-e64e-49ed-b41b-15a7e9fdd029","Type":"ContainerDied","Data":"2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d"} Nov 02 08:18:43.525684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.525359 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm" event={"ID":"cfdc734b-e64e-49ed-b41b-15a7e9fdd029","Type":"ContainerDied","Data":"c1702b2535ea5fa79cf453f3f3675d91a86da10a80ade695604a7577a90c181b"} Nov 02 08:18:43.525684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.525395 5055 scope.go:122] "RemoveContainer" containerID="2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d" Nov 02 08:18:43.536740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.536714 5055 scope.go:122] "RemoveContainer" containerID="fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa" Nov 02 08:18:43.546685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.546663 5055 scope.go:122] "RemoveContainer" containerID="2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d" Nov 02 08:18:43.547026 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:43.547002 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d\": container with ID starting with 2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d not found: ID does not exist" containerID="2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d" Nov 02 08:18:43.547094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.547034 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d"} err="failed to get container status \"2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d\": rpc error: code = NotFound desc = could not find container \"2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d\": container with ID starting with 2ddb10c48450335a8440027008cb8eb1532be72391f8eb75dd1218618214aa6d not found: ID does not exist" Nov 02 08:18:43.547094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.547056 5055 scope.go:122] "RemoveContainer" containerID="fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa" Nov 02 08:18:43.547394 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:43.547376 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa\": container with ID starting with fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa not found: ID does not exist" containerID="fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa" Nov 02 08:18:43.547454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.547400 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa"} err="failed to get container status \"fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa\": rpc error: code = NotFound desc = could not find container \"fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa\": container with ID starting with fb6543f29e2d8fed13cfb04603a968ccd8723492224db65f6c0422d82724ccaa not found: ID does not exist" Nov 02 08:18:43.560766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.560731 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.560952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.560782 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.560952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.560811 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.560952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.560851 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.560952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.560881 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.560952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.560935 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561018 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561092 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561177 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert\") pod \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\" (UID: \"cfdc734b-e64e-49ed-b41b-15a7e9fdd029\") " Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561181 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561116 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561229 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:43.561250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561242 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:43.561602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561576 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:43.561705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561686 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.561745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561710 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.561745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561725 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.561745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561739 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.561831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.561753 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.563479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.563449 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:43.563595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.563514 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:43.563683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.563666 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:18:43.563717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.563691 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq" pod "cfdc734b-e64e-49ed-b41b-15a7e9fdd029" (UID: "cfdc734b-e64e-49ed-b41b-15a7e9fdd029"). InnerVolumeSpecName "kube-api-access-66jfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:43.663583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.663505 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jfq\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-kube-api-access-66jfq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.663583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.663559 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.663583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.663574 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.663583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.663587 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfdc734b-e64e-49ed-b41b-15a7e9fdd029-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:43.938902 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.938864 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm"] Nov 02 08:18:43.942410 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:43.942380 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2b62c067-openshift-default-f85b48b9c-pb4vm"] Nov 02 08:18:44.257202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:44.257164 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2"] Nov 02 08:18:44.381274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:44.381240 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:18:45.092337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:45.092304 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" path="/var/lib/kubelet/pods/cfdc734b-e64e-49ed-b41b-15a7e9fdd029/volumes" Nov 02 08:18:49.257937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.257891 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" podUID="41c971b3-feda-4799-9c1d-ea133ca67d29" containerName="payload-pre-processing" containerID="cri-o://ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3" gracePeriod=25 Nov 02 08:18:49.545118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.545094 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:18:49.571483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.571451 5055 generic.go:356] "Generic (PLEG): container finished" podID="41c971b3-feda-4799-9c1d-ea133ca67d29" containerID="ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3" exitCode=0 Nov 02 08:18:49.571644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.571514 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" Nov 02 08:18:49.571644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.571533 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" event={"ID":"41c971b3-feda-4799-9c1d-ea133ca67d29","Type":"ContainerDied","Data":"ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3"} Nov 02 08:18:49.571644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.571611 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2" event={"ID":"41c971b3-feda-4799-9c1d-ea133ca67d29","Type":"ContainerDied","Data":"16544e3a0f1672bf8359a4ccf7045dbf8a71111aba0c2551fd469043c69687f5"} Nov 02 08:18:49.571644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.571635 5055 scope.go:122] "RemoveContainer" containerID="ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3" Nov 02 08:18:49.581003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.580980 5055 scope.go:122] "RemoveContainer" containerID="ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3" Nov 02 08:18:49.581355 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:49.581333 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3\": container with ID starting with ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3 not found: ID does not exist" containerID="ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3" Nov 02 08:18:49.581445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.581366 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3"} err="failed to get container status \"ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3\": rpc error: code = NotFound desc = could not find container \"ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3\": container with ID starting with ea2c880ad733985c45bc678f2ff76164cbafc7156fd3e804bfd5676da9cf64d3 not found: ID does not exist" Nov 02 08:18:49.629463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.629431 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume\") pod \"41c971b3-feda-4799-9c1d-ea133ca67d29\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " Nov 02 08:18:49.629665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.629479 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t\" (UniqueName: \"kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t\") pod \"41c971b3-feda-4799-9c1d-ea133ca67d29\" (UID: \"41c971b3-feda-4799-9c1d-ea133ca67d29\") " Nov 02 08:18:49.629894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.629836 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume" pod "41c971b3-feda-4799-9c1d-ea133ca67d29" (UID: "41c971b3-feda-4799-9c1d-ea133ca67d29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:49.631867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.631843 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t" pod "41c971b3-feda-4799-9c1d-ea133ca67d29" (UID: "41c971b3-feda-4799-9c1d-ea133ca67d29"). InnerVolumeSpecName "kube-api-access-dh95t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:49.731303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.731270 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh95t\" (UniqueName: \"kubernetes.io/projected/41c971b3-feda-4799-9c1d-ea133ca67d29-kube-api-access-dh95t\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:49.731303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.731298 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c971b3-feda-4799-9c1d-ea133ca67d29-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:49.987109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.987077 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2"] Nov 02 08:18:49.990464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:49.990433 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16457c0e-d4c6c5d59-6hzt2"] Nov 02 08:18:50.163769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.163678 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6"] Nov 02 08:18:50.164174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164157 5055 cpu_manager.go:414] "containerMap: removing container" podUID="41c971b3-feda-4799-9c1d-ea133ca67d29" containerName="payload-pre-processing" Nov 02 08:18:50.164215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164176 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c971b3-feda-4799-9c1d-ea133ca67d29" containerName="payload-pre-processing" Nov 02 08:18:50.164215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164193 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" containerName="revoke-keys" Nov 02 08:18:50.164215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164198 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" containerName="revoke-keys" Nov 02 08:18:50.164215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164209 5055 cpu_manager.go:414] "containerMap: removing container" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" Nov 02 08:18:50.164215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164214 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" Nov 02 08:18:50.164385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164313 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c971b3-feda-4799-9c1d-ea133ca67d29" containerName="payload-pre-processing" Nov 02 08:18:50.164385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164320 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" containerName="revoke-keys" Nov 02 08:18:50.164385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164326 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" Nov 02 08:18:50.164385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.164341 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" Nov 02 08:18:50.167682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.167663 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.255799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.255741 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-108cf9-openshift-default-dockercfg-gprcr\"" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256383 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256523 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256581 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256613 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256695 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256759 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256796 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256832 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.258237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.256919 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.261302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.261279 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6"] Nov 02 08:18:50.358526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358485 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358534 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358566 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358637 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358679 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358804 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358874 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358908 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.358993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.358986 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.359164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.359114 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.359281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.359258 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.359363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.359320 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.359405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.359368 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.359590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.359571 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.361089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.361065 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.361443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.361421 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.415617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.415533 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.416128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.416099 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.480774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.480734 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:50.903474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.903449 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6"] Nov 02 08:18:50.905039 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:18:50.905001 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9120df_a37d_4d38_b323_4ed820bbd0a0.slice/crio-3300d13e0fa7d8d95078969f73d122dee0dccbae74527e9aed49385c861fe5d3 WatchSource:0}: Error finding container 3300d13e0fa7d8d95078969f73d122dee0dccbae74527e9aed49385c861fe5d3: Status 404 returned error can't find the container with id 3300d13e0fa7d8d95078969f73d122dee0dccbae74527e9aed49385c861fe5d3 Nov 02 08:18:50.907537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.907498 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:50.907636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.907584 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:50.907636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:50.907630 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:18:51.092714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:51.092683 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c971b3-feda-4799-9c1d-ea133ca67d29" path="/var/lib/kubelet/pods/41c971b3-feda-4799-9c1d-ea133ca67d29/volumes" Nov 02 08:18:51.590540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:51.590501 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" event={"ID":"7f9120df-a37d-4d38-b323-4ed820bbd0a0","Type":"ContainerStarted","Data":"178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf"} Nov 02 08:18:51.591050 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:51.590571 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" event={"ID":"7f9120df-a37d-4d38-b323-4ed820bbd0a0","Type":"ContainerStarted","Data":"3300d13e0fa7d8d95078969f73d122dee0dccbae74527e9aed49385c861fe5d3"} Nov 02 08:18:51.698167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:51.698122 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d"] Nov 02 08:18:51.742430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:51.742359 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podStartSLOduration=1.742338675 podStartE2EDuration="1.742338675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:51.698771861 +0000 UTC m=+778.789392279" watchObservedRunningTime="2026-11-02 08:18:51.742338675 +0000 UTC m=+778.832959098" Nov 02 08:18:52.481706 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:52.481666 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:52.487286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:52.487258 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:52.598106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:52.598066 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:52.599146 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:52.599112 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:18:54.932629 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:54.932585 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:18:56.699080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:56.699037 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" podUID="8f3b7cc4-0147-4311-b394-44a63a6aa970" containerName="payload-processing" containerID="cri-o://6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1" gracePeriod=25 Nov 02 08:18:56.972187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:56.972162 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:18:57.044204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.044172 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd\" (UniqueName: \"kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd\") pod \"8f3b7cc4-0147-4311-b394-44a63a6aa970\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " Nov 02 08:18:57.044398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.044268 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume\") pod \"8f3b7cc4-0147-4311-b394-44a63a6aa970\" (UID: \"8f3b7cc4-0147-4311-b394-44a63a6aa970\") " Nov 02 08:18:57.044631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.044604 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume" pod "8f3b7cc4-0147-4311-b394-44a63a6aa970" (UID: "8f3b7cc4-0147-4311-b394-44a63a6aa970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:57.046367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.046337 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd" pod "8f3b7cc4-0147-4311-b394-44a63a6aa970" (UID: "8f3b7cc4-0147-4311-b394-44a63a6aa970"). InnerVolumeSpecName "kube-api-access-c2pkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:57.145395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.145362 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2pkd\" (UniqueName: \"kubernetes.io/projected/8f3b7cc4-0147-4311-b394-44a63a6aa970-kube-api-access-c2pkd\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:57.145395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.145391 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b7cc4-0147-4311-b394-44a63a6aa970-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:57.630723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.630687 5055 generic.go:356] "Generic (PLEG): container finished" podID="8f3b7cc4-0147-4311-b394-44a63a6aa970" containerID="6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1" exitCode=0 Nov 02 08:18:57.630947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.630742 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" Nov 02 08:18:57.630947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.630775 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" event={"ID":"8f3b7cc4-0147-4311-b394-44a63a6aa970","Type":"ContainerDied","Data":"6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1"} Nov 02 08:18:57.630947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.630839 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d" event={"ID":"8f3b7cc4-0147-4311-b394-44a63a6aa970","Type":"ContainerDied","Data":"57e0d19e9e2354144038338779dd8350706cbbf0ed399c66b8e45df145946277"} Nov 02 08:18:57.630947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.630868 5055 scope.go:122] "RemoveContainer" containerID="6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1" Nov 02 08:18:57.640411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.640388 5055 scope.go:122] "RemoveContainer" containerID="6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1" Nov 02 08:18:57.640723 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:18:57.640703 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1\": container with ID starting with 6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1 not found: ID does not exist" containerID="6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1" Nov 02 08:18:57.640780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.640732 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1"} err="failed to get container status \"6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1\": rpc error: code = NotFound desc = could not find container \"6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1\": container with ID starting with 6b024381a2334cbed8337e0b4a5378d786b796c18da7cefcf9e3a267be165de1 not found: ID does not exist" Nov 02 08:18:57.784278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.784239 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d"] Nov 02 08:18:57.830020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.829976 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e734c1c-7546bf48b7-fs62d"] Nov 02 08:18:57.981521 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.981484 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc"] Nov 02 08:18:57.994045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:57.994006 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-16457c0e-kg2wc"] Nov 02 08:18:59.092080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.092046 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3b7cc4-0147-4311-b394-44a63a6aa970" path="/var/lib/kubelet/pods/8f3b7cc4-0147-4311-b394-44a63a6aa970/volumes" Nov 02 08:18:59.092526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.092509 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903750b3-bdba-4e51-9066-82fb40d7d87e" path="/var/lib/kubelet/pods/903750b3-bdba-4e51-9066-82fb40d7d87e/volumes" Nov 02 08:18:59.411332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.411230 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc"] Nov 02 08:18:59.412078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.412054 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8f3b7cc4-0147-4311-b394-44a63a6aa970" containerName="payload-processing" Nov 02 08:18:59.412243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.412079 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3b7cc4-0147-4311-b394-44a63a6aa970" containerName="payload-processing" Nov 02 08:18:59.412243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.412122 5055 cpu_manager.go:414] "containerMap: removing container" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" Nov 02 08:18:59.412243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.412131 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdc734b-e64e-49ed-b41b-15a7e9fdd029" containerName="istio-proxy" Nov 02 08:18:59.412403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.412273 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3b7cc4-0147-4311-b394-44a63a6aa970" containerName="payload-processing" Nov 02 08:18:59.432228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.432195 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.482009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.481974 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-096685-openshift-default-dockercfg-fcvv8\"" Nov 02 08:18:59.549163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.545285 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc"] Nov 02 08:18:59.590768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.590726 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.590981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.590827 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.590981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.590893 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.590981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.590912 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.590981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.590931 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.591224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.591036 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.591224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.591075 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8fg4\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.591224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.591164 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.591224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.591203 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.694962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.694853 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695230 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695321 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695348 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695372 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695429 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695479 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8fg4\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695524 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.695609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.695553 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.696054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.696031 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.696622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.696601 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.696990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.696966 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.699451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.698636 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.699794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.699653 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.701061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.701037 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.701397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.701378 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.750904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.750863 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8fg4\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.751817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.751780 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:18:59.761659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:18:59.761631 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:19:00.182182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.182135 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc"] Nov 02 08:19:00.183412 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:19:00.183381 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5653ed7_4d7c_4f4a_9c24_3d881beeea94.slice/crio-d077e2ca2d8872a333ac1279abc97af423ed9d056c29cd577cab7e3a6c3c2564 WatchSource:0}: Error finding container d077e2ca2d8872a333ac1279abc97af423ed9d056c29cd577cab7e3a6c3c2564: Status 404 returned error can't find the container with id d077e2ca2d8872a333ac1279abc97af423ed9d056c29cd577cab7e3a6c3c2564 Nov 02 08:19:00.186128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.186092 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:00.186246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.186184 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:00.186246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.186218 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:00.681743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.681706 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" event={"ID":"a5653ed7-4d7c-4f4a-9c24-3d881beeea94","Type":"ContainerStarted","Data":"6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec"} Nov 02 08:19:00.681960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.681764 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" event={"ID":"a5653ed7-4d7c-4f4a-9c24-3d881beeea94","Type":"ContainerStarted","Data":"d077e2ca2d8872a333ac1279abc97af423ed9d056c29cd577cab7e3a6c3c2564"} Nov 02 08:19:00.762756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.762718 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:19:00.793972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:00.793913 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podStartSLOduration=1.793898729 podStartE2EDuration="1.793898729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:00.791591072 +0000 UTC m=+787.882211490" watchObservedRunningTime="2026-11-02 08:19:00.793898729 +0000 UTC m=+787.884519213" Nov 02 08:19:01.767112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:01.767083 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:19:02.696406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:02.696374 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:19:02.697539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:02.697510 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:19:06.060482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:06.060444 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:19:16.811625 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:16.811574 5055 generic.go:356] "Generic (PLEG): container finished" podID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerID="dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb" exitCode=0 Nov 02 08:19:16.812332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:16.811634 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" event={"ID":"88c2b98a-18f5-4b2e-a54a-ce702a109380","Type":"ContainerDied","Data":"dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb"} Nov 02 08:19:16.812332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:16.812107 5055 scope.go:122] "RemoveContainer" containerID="dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb" Nov 02 08:19:16.813575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:16.813541 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:16.813677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:16.813614 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:16.813677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:16.813645 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:17.820385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:17.820351 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" event={"ID":"88c2b98a-18f5-4b2e-a54a-ce702a109380","Type":"ContainerStarted","Data":"1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde"} Nov 02 08:19:18.372209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:18.372171 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:19:18.373567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:18.373540 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 08:19:18.373680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:18.373593 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 08:19:19.343829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:19.343794 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:19:19.372617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:19.372206 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 08:19:19.372617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:19.372275 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 08:19:20.371721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:20.371689 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 08:19:20.372263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:20.372236 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 08:19:21.372385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:21.372349 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 08:19:21.372769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:21.372418 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 08:19:22.372533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:22.372494 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 08:19:22.373069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:22.372564 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 08:19:23.427382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:23.427307 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:19:23.428635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:23.428608 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:19:24.923705 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:19:24.923668 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:19:29.335691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:29.335653 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m"] Nov 02 08:19:29.381462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:29.381425 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e734c1c-26w8m"] Nov 02 08:19:30.693000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:30.692951 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg"] Nov 02 08:19:30.693551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:30.693179 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" containerID="cri-o://760ea10b27142095bef8ad0ee2cdaedcbca4502b2e52d1cce8a69b3cfcb7b336" gracePeriod=30 Nov 02 08:19:30.814994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:30.810208 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:19:31.091642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:31.091603 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19" path="/var/lib/kubelet/pods/4b1de7ff-0a18-4ff0-8b7d-c2f5e006fb19/volumes" Nov 02 08:19:32.140567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.140523 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5"] Nov 02 08:19:32.146057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.146023 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.229134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.229089 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5"] Nov 02 08:19:32.260873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.260839 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8e6407-openshift-default-dockercfg-lcnm5\"" Nov 02 08:19:32.328868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.328813 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.328900 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.328936 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.329005 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.329075 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.329108 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.329185 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.329225 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9h4\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.329371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.329269 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.430880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.430764 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.430880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.430829 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.430894 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.430959 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.431004 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.431062 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.431121 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9h4\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.431198 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.431270 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.431733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.431705 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.432035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.432010 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.432344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.432320 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.434189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.434159 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.439297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.435344 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.443424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.443389 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.444761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.444731 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.512760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.512720 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9h4\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.512760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.512743 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:32.760901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:32.760827 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:33.229198 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:33.228397 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5"] Nov 02 08:19:33.261164 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:19:33.257854 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2211da_e856_427e_946c_6c7b14714377.slice/crio-7cb68a4273b2cec99dfa8b787091346bf655b27ad6dafc2aabfc2c4097c3413e WatchSource:0}: Error finding container 7cb68a4273b2cec99dfa8b787091346bf655b27ad6dafc2aabfc2c4097c3413e: Status 404 returned error can't find the container with id 7cb68a4273b2cec99dfa8b787091346bf655b27ad6dafc2aabfc2c4097c3413e Nov 02 08:19:33.281413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:33.273965 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:33.281413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:33.274061 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:33.281413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:33.274107 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:33.958509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:33.958461 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" event={"ID":"3b2211da-e856-427e-946c-6c7b14714377","Type":"ContainerStarted","Data":"285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8"} Nov 02 08:19:33.958730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:33.958526 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" event={"ID":"3b2211da-e856-427e-946c-6c7b14714377","Type":"ContainerStarted","Data":"7cb68a4273b2cec99dfa8b787091346bf655b27ad6dafc2aabfc2c4097c3413e"} Nov 02 08:19:34.152554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:34.152503 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podStartSLOduration=2.152488203 podStartE2EDuration="2.152488203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:34.112676855 +0000 UTC m=+821.203297273" watchObservedRunningTime="2026-11-02 08:19:34.152488203 +0000 UTC m=+821.243108622" Nov 02 08:19:34.761857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:34.761817 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:34.769438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:34.769407 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:34.964816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:34.964780 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:34.965724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:34.965704 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:19:35.972573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:35.972491 5055 generic.go:356] "Generic (PLEG): container finished" podID="2d06009c-3094-4803-aa06-c283ff522cc7" containerID="760ea10b27142095bef8ad0ee2cdaedcbca4502b2e52d1cce8a69b3cfcb7b336" exitCode=0 Nov 02 08:19:35.973054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:35.972572 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" event={"ID":"2d06009c-3094-4803-aa06-c283ff522cc7","Type":"ContainerDied","Data":"760ea10b27142095bef8ad0ee2cdaedcbca4502b2e52d1cce8a69b3cfcb7b336"} Nov 02 08:19:35.973054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:35.972647 5055 scope.go:122] "RemoveContainer" containerID="b0a96b1053238b62f0c08a717f4746327098a098a3a010723481181568f86424" Nov 02 08:19:36.435325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.435298 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503500 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503571 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503609 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503648 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503736 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503770 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503831 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503885 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.503989 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket\") pod \"2d06009c-3094-4803-aa06-c283ff522cc7\" (UID: \"2d06009c-3094-4803-aa06-c283ff522cc7\") " Nov 02 08:19:36.505849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.505458 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:36.505849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.505835 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:36.506091 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.506062 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:36.506268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.506096 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:19:36.506432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.506380 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:36.508192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.508162 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:36.508469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.508443 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:19:36.508469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.508459 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "kube-api-access-gznx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:19:36.508707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.508690 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo" pod "2d06009c-3094-4803-aa06-c283ff522cc7" (UID: "2d06009c-3094-4803-aa06-c283ff522cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:19:36.605362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.605311 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610030 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610241 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gznx8\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-kube-api-access-gznx8\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610334 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d06009c-3094-4803-aa06-c283ff522cc7-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610407 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610478 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d06009c-3094-4803-aa06-c283ff522cc7-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610552 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610621 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d06009c-3094-4803-aa06-c283ff522cc7-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.610700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.610690 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d06009c-3094-4803-aa06-c283ff522cc7-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:36.986166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.983379 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" Nov 02 08:19:36.999490 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.996584 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg" event={"ID":"2d06009c-3094-4803-aa06-c283ff522cc7","Type":"ContainerDied","Data":"9a76ae44079f24a52cf346d68c15ab99cb169d3e6cb975901d4ee6720ff7a1b5"} Nov 02 08:19:36.999490 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:36.996704 5055 scope.go:122] "RemoveContainer" containerID="760ea10b27142095bef8ad0ee2cdaedcbca4502b2e52d1cce8a69b3cfcb7b336" Nov 02 08:19:37.163234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:37.163200 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg"] Nov 02 08:19:37.168876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:37.168843 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2e734c1c-openshift-default-64c45c6cd9-rhsjg"] Nov 02 08:19:39.092774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:39.092737 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" path="/var/lib/kubelet/pods/2d06009c-3094-4803-aa06-c283ff522cc7/volumes" Nov 02 08:19:42.043093 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:42.043054 5055 generic.go:356] "Generic (PLEG): container finished" podID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerID="178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf" exitCode=0 Nov 02 08:19:42.043566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:42.043121 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" event={"ID":"7f9120df-a37d-4d38-b323-4ed820bbd0a0","Type":"ContainerDied","Data":"178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf"} Nov 02 08:19:42.043612 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:42.043571 5055 scope.go:122] "RemoveContainer" containerID="178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf" Nov 02 08:19:42.044995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:42.044957 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:42.045203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:42.045040 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:42.045203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:42.045088 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:43.051066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:43.051029 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" event={"ID":"7f9120df-a37d-4d38-b323-4ed820bbd0a0","Type":"ContainerStarted","Data":"5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719"} Nov 02 08:19:43.497947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:43.497906 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:19:43.501031 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:43.500988 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 08:19:43.501421 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:43.501386 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 08:19:44.493569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:44.493527 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 08:19:44.494225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:44.494189 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 08:19:45.482023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:45.481988 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 08:19:45.482369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:45.482325 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 08:19:46.486763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:46.486722 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 08:19:46.487459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:46.487420 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 08:19:47.411821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:47.411776 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:19:47.488166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:47.487293 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 08:19:47.488166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:47.487358 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 08:19:48.509378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:48.509327 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:19:48.514080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:48.511308 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:19:48.515744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:48.515705 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:19:54.921596 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:19:54.921559 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:19:57.166071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:57.166028 5055 generic.go:356] "Generic (PLEG): container finished" podID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerID="6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec" exitCode=0 Nov 02 08:19:57.166650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:57.166099 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" event={"ID":"a5653ed7-4d7c-4f4a-9c24-3d881beeea94","Type":"ContainerDied","Data":"6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec"} Nov 02 08:19:57.166650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:57.166613 5055 scope.go:122] "RemoveContainer" containerID="6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec" Nov 02 08:19:57.168048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:57.168021 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:57.168128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:57.168087 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:57.168128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:57.168118 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:19:58.173955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:58.173917 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" event={"ID":"a5653ed7-4d7c-4f4a-9c24-3d881beeea94","Type":"ContainerStarted","Data":"375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f"} Nov 02 08:19:58.762557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:58.762517 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:19:58.764039 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:58.764008 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 08:19:58.764186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:58.764061 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 08:19:59.762235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:59.762200 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 08:19:59.762625 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:19:59.762253 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 08:20:00.762620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:00.762581 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 08:20:00.763158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:00.762758 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 08:20:01.762337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:01.762299 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 08:20:01.762514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:01.762378 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 08:20:02.447644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.447599 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb"] Nov 02 08:20:02.448188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.448168 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" Nov 02 08:20:02.448257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.448191 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" Nov 02 08:20:02.448257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.448211 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" Nov 02 08:20:02.448257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.448217 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" Nov 02 08:20:02.448358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.448333 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" Nov 02 08:20:02.448358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.448345 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d06009c-3094-4803-aa06-c283ff522cc7" containerName="istio-proxy" Nov 02 08:20:02.452021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.451995 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.522533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.522496 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb"] Nov 02 08:20:02.548772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.548734 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"default-dockercfg-d8n5d\"" Nov 02 08:20:02.548772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.548753 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"kube-root-ca.crt\"" Nov 02 08:20:02.548772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.548774 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"simulator-w2-10d2fb-kserve-self-signed-certs\"" Nov 02 08:20:02.549032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.548790 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"openshift-service-ca.crt\"" Nov 02 08:20:02.559453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.559425 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.559597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.559492 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.559597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.559585 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.559703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.559637 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.559703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.559661 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4flv\" (UniqueName: \"kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.559831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.559814 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.660511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.660471 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.660726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.660535 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.660726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.660561 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.660726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.660595 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.660726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.660612 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4flv\" (UniqueName: \"kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.660726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.660675 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.661096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.661070 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.661217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.661081 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.661217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.661133 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.662917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.662893 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.663108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.663093 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.706189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.703249 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:20:02.713489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.713460 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4flv\" (UniqueName: \"kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv\") pod \"simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:02.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.762130 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 08:20:02.762396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.762227 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 08:20:02.765356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:02.765332 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:03.167413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:03.167386 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb"] Nov 02 08:20:03.169471 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:20:03.169437 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f512082_1172_433d_9043_181b9895b517.slice/crio-fb0f1dcd1704bf50bbedf818dbaeac5ca89a0b85157bbc3c9a0bdb02dd565aea WatchSource:0}: Error finding container fb0f1dcd1704bf50bbedf818dbaeac5ca89a0b85157bbc3c9a0bdb02dd565aea: Status 404 returned error can't find the container with id fb0f1dcd1704bf50bbedf818dbaeac5ca89a0b85157bbc3c9a0bdb02dd565aea Nov 02 08:20:03.228403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:03.228372 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" event={"ID":"2f512082-1172-433d-9043-181b9895b517","Type":"ContainerStarted","Data":"fb0f1dcd1704bf50bbedf818dbaeac5ca89a0b85157bbc3c9a0bdb02dd565aea"} Nov 02 08:20:03.947802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:03.947709 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:20:03.949492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:03.949464 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:20:04.239681 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:04.239642 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" event={"ID":"2f512082-1172-433d-9043-181b9895b517","Type":"ContainerStarted","Data":"b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148"} Nov 02 08:20:04.350662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:04.350608 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podStartSLOduration=2.350595465 podStartE2EDuration="2.350595465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:04.348812817 +0000 UTC m=+851.439433237" watchObservedRunningTime="2026-11-02 08:20:04.350595465 +0000 UTC m=+851.441215881" Nov 02 08:20:04.766255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:04.766216 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:07.008893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:07.008855 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:20:10.145706 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.145668 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn"] Nov 02 08:20:10.149672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.149651 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.201894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.201715 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9d77b6\"/\"kube-root-ca.crt\"" Nov 02 08:20:10.247605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.247570 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9d77b6\"/\"openshift-service-ca.crt\"" Nov 02 08:20:10.247605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.247573 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9d77b6\"/\"default-dockercfg-gvtss\"" Nov 02 08:20:10.247996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.247976 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9d77b6\"/\"simulator-w5-9d77b6-kserve-self-signed-certs\"" Nov 02 08:20:10.250654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.250630 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn"] Nov 02 08:20:10.273346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.273318 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.273456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.273348 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.273456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.273380 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nck89\" (UniqueName: \"kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.273565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.273546 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.273616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.273601 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.273726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.273713 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375453 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375608 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375642 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375662 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375688 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nck89\" (UniqueName: \"kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375813 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.375975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375947 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.376035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.375975 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.376092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.376056 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.378180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.378136 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.378462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.378441 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.432263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.432186 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nck89\" (UniqueName: \"kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89\") pod \"simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.464065 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.464032 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:10.852417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:10.852393 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn"] Nov 02 08:20:10.854255 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:20:10.854224 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98c909df_aff4_4780_acf8_06d780b10794.slice/crio-cf1669f45785cbe33f3555a1ebd55947b9aae2f2e36abafafb57bf1915523753 WatchSource:0}: Error finding container cf1669f45785cbe33f3555a1ebd55947b9aae2f2e36abafafb57bf1915523753: Status 404 returned error can't find the container with id cf1669f45785cbe33f3555a1ebd55947b9aae2f2e36abafafb57bf1915523753 Nov 02 08:20:11.286580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:11.286541 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" event={"ID":"98c909df-aff4-4780-acf8-06d780b10794","Type":"ContainerStarted","Data":"88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02"} Nov 02 08:20:11.286580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:11.286587 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" event={"ID":"98c909df-aff4-4780-acf8-06d780b10794","Type":"ContainerStarted","Data":"cf1669f45785cbe33f3555a1ebd55947b9aae2f2e36abafafb57bf1915523753"} Nov 02 08:20:11.396339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:11.396285 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podStartSLOduration=1.396269691 podStartE2EDuration="1.396269691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:11.393534858 +0000 UTC m=+858.484155301" watchObservedRunningTime="2026-11-02 08:20:11.396269691 +0000 UTC m=+858.486890109" Nov 02 08:20:11.464834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:11.464803 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:12.766272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:12.766227 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:12.779533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:12.779502 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:13.310447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:13.310419 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:20:20.464304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:20.464275 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:20.477214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:20.477186 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:21.374412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:21.374381 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:20:24.922369 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:20:24.922277 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:20:28.420453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.420419 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs"] Nov 02 08:20:28.426213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.426187 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.478736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.478699 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs"] Nov 02 08:20:28.527394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.522087 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:20:28.527394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.522206 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:20:28.527394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.522241 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:20:28.527394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.522280 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:20:28.535416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.535369 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.535623 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.535562 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22pj\" (UniqueName: \"kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.535943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.535919 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.637295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.637246 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.637544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.637339 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22pj\" (UniqueName: \"kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.637544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.637458 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.638044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.638014 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.639945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.639923 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.699222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.699023 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22pj\" (UniqueName: \"kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj\") pod \"maas-api-revoke-sub-b4437474-lgdcs\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:28.743546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:28.743504 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:29.151055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:29.151031 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs"] Nov 02 08:20:29.153007 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:20:29.152980 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68736ea4_2030_44d9_be5e_b9fe8832c6d7.slice/crio-434226ed800055ed8f0ee203f170d199dd19639e18cd15c493452d595415c557 WatchSource:0}: Error finding container 434226ed800055ed8f0ee203f170d199dd19639e18cd15c493452d595415c557: Status 404 returned error can't find the container with id 434226ed800055ed8f0ee203f170d199dd19639e18cd15c493452d595415c557 Nov 02 08:20:29.433221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:29.433170 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" event={"ID":"68736ea4-2030-44d9-be5e-b9fe8832c6d7","Type":"ContainerStarted","Data":"434226ed800055ed8f0ee203f170d199dd19639e18cd15c493452d595415c557"} Nov 02 08:20:30.441335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:30.441299 5055 generic.go:356] "Generic (PLEG): container finished" podID="68736ea4-2030-44d9-be5e-b9fe8832c6d7" containerID="912eb4cbd408be863be0cd8a2b6d217f097da0eba2ba677829fcacedbf3110c9" exitCode=0 Nov 02 08:20:30.441738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:30.441382 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" event={"ID":"68736ea4-2030-44d9-be5e-b9fe8832c6d7","Type":"ContainerDied","Data":"912eb4cbd408be863be0cd8a2b6d217f097da0eba2ba677829fcacedbf3110c9"} Nov 02 08:20:31.606122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.606043 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:31.673836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.673806 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca\") pod \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " Nov 02 08:20:31.674011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.673859 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj\" (UniqueName: \"kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj\") pod \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " Nov 02 08:20:31.674011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.673957 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca\") pod \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\" (UID: \"68736ea4-2030-44d9-be5e-b9fe8832c6d7\") " Nov 02 08:20:31.674253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.674229 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca" pod "68736ea4-2030-44d9-be5e-b9fe8832c6d7" (UID: "68736ea4-2030-44d9-be5e-b9fe8832c6d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:20:31.674563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.674530 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:31.676180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.676134 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj" pod "68736ea4-2030-44d9-be5e-b9fe8832c6d7" (UID: "68736ea4-2030-44d9-be5e-b9fe8832c6d7"). InnerVolumeSpecName "kube-api-access-r22pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:20:31.676275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.676154 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca" pod "68736ea4-2030-44d9-be5e-b9fe8832c6d7" (UID: "68736ea4-2030-44d9-be5e-b9fe8832c6d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:20:31.775974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.775935 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68736ea4-2030-44d9-be5e-b9fe8832c6d7-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:31.775974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:31.775969 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22pj\" (UniqueName: \"kubernetes.io/projected/68736ea4-2030-44d9-be5e-b9fe8832c6d7-kube-api-access-r22pj\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:32.455166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:32.455109 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" Nov 02 08:20:32.455342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:32.455105 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs" event={"ID":"68736ea4-2030-44d9-be5e-b9fe8832c6d7","Type":"ContainerDied","Data":"434226ed800055ed8f0ee203f170d199dd19639e18cd15c493452d595415c557"} Nov 02 08:20:32.455342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:32.455277 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="434226ed800055ed8f0ee203f170d199dd19639e18cd15c493452d595415c557" Nov 02 08:20:34.966320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:34.966286 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 08:20:34.966748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:34.966340 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 08:20:35.475048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:35.475013 5055 generic.go:356] "Generic (PLEG): container finished" podID="3b2211da-e856-427e-946c-6c7b14714377" containerID="285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8" exitCode=0 Nov 02 08:20:35.475243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:35.475064 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" event={"ID":"3b2211da-e856-427e-946c-6c7b14714377","Type":"ContainerDied","Data":"285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8"} Nov 02 08:20:35.475517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:35.475497 5055 scope.go:122] "RemoveContainer" containerID="285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8" Nov 02 08:20:35.476949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:35.476917 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:20:35.477051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:35.476987 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:20:35.477051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:35.477018 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:20:36.498021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:36.497984 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" event={"ID":"3b2211da-e856-427e-946c-6c7b14714377","Type":"ContainerStarted","Data":"7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794"} Nov 02 08:20:36.761502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:36.761406 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:20:36.761686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:36.761620 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 08:20:36.761686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:36.761672 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 08:20:37.761854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:37.761816 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 08:20:37.762313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:37.761870 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 08:20:38.762095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:38.762050 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 08:20:38.762544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:38.762116 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 08:20:39.761933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:39.761889 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 08:20:39.762121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:39.761961 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 08:20:39.998926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:39.998880 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:20:40.447818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.447783 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf"] Nov 02 08:20:40.448325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.448310 5055 cpu_manager.go:414] "containerMap: removing container" podUID="68736ea4-2030-44d9-be5e-b9fe8832c6d7" containerName="revoke-keys" Nov 02 08:20:40.448376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.448326 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68736ea4-2030-44d9-be5e-b9fe8832c6d7" containerName="revoke-keys" Nov 02 08:20:40.448425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.448416 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="68736ea4-2030-44d9-be5e-b9fe8832c6d7" containerName="revoke-keys" Nov 02 08:20:40.450821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.450801 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.499607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.499571 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-096685\"/\"default-dockercfg-rqt6f\"" Nov 02 08:20:40.547598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.547518 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-096685\"/\"openshift-service-ca.crt\"" Nov 02 08:20:40.547905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.547642 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-096685\"/\"kube-root-ca.crt\"" Nov 02 08:20:40.551334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.551267 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf"] Nov 02 08:20:40.604174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.580276 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.604174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.580382 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.604174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.580460 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.604174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.580488 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfq6\" (UniqueName: \"kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.604174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.580561 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.604174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.580623 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.605947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.605888 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-096685\"/\"simulator-w6-096685-kserve-self-signed-certs\"" Nov 02 08:20:40.683170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.682826 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.683170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.682907 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.683170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.682953 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.683170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.682978 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfq6\" (UniqueName: \"kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.683170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.683034 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.683170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.683081 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.692422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.684395 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.692422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.685675 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.692422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.691183 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.717777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.717743 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.718804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.718771 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.744102 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.744067 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfq6\" (UniqueName: \"kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6\") pod \"simulator-w6-096685-kserve-66b84844cc-qb6tf\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.761853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.761824 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:40.762031 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.761967 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 08:20:40.762118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:40.762032 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 08:20:41.164065 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.164036 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf"] Nov 02 08:20:41.165478 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:20:41.165447 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdde112c_cddd_479e_ada5_2535c5a9ea39.slice/crio-dbfbfb2eb41de212912e1d574406cfd8a8fc6888d104f478ae524361e0ec2e79 WatchSource:0}: Error finding container dbfbfb2eb41de212912e1d574406cfd8a8fc6888d104f478ae524361e0ec2e79: Status 404 returned error can't find the container with id dbfbfb2eb41de212912e1d574406cfd8a8fc6888d104f478ae524361e0ec2e79 Nov 02 08:20:41.555886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.555833 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" event={"ID":"bdde112c-cddd-479e-ada5-2535c5a9ea39","Type":"ContainerStarted","Data":"b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7"} Nov 02 08:20:41.556119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.555895 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" event={"ID":"bdde112c-cddd-479e-ada5-2535c5a9ea39","Type":"ContainerStarted","Data":"dbfbfb2eb41de212912e1d574406cfd8a8fc6888d104f478ae524361e0ec2e79"} Nov 02 08:20:41.662576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.662512 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podStartSLOduration=1.662498318 podStartE2EDuration="1.662498318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:41.66187167 +0000 UTC m=+888.752492089" watchObservedRunningTime="2026-11-02 08:20:41.662498318 +0000 UTC m=+888.753118752" Nov 02 08:20:41.761893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.761858 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:41.765797 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.765771 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:20:41.766266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.766244 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:20:41.766945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:41.766927 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:20:50.762504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:50.762464 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:50.775416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:50.775383 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:51.639433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.639398 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:20:51.660928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.660885 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4"] Nov 02 08:20:51.664697 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.664670 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.717721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.717669 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:20:51.733506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.733469 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:20:51.733723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.733528 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:20:51.737370 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.737339 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:20:51.738535 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.738508 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4"] Nov 02 08:20:51.749811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.749774 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgzg\" (UniqueName: \"kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.749993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.749839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.749993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.749875 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.854168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.851348 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.854168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.851597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgzg\" (UniqueName: \"kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.854168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.851702 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.856825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.855440 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.856825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.856123 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.910954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.910868 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgzg\" (UniqueName: \"kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg\") pod \"maas-api-revoke-sub-ee09c02a-bmhb4\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:51.983175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:51.983117 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:52.363217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:52.363182 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4"] Nov 02 08:20:52.365942 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:20:52.365900 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ca3f51_620a_47c2_ab1c_4146bc3e08fa.slice/crio-7a20065f0a641e8ba694230c022967ec35385b49ad3997e9550a1fe3a3c7cfd5 WatchSource:0}: Error finding container 7a20065f0a641e8ba694230c022967ec35385b49ad3997e9550a1fe3a3c7cfd5: Status 404 returned error can't find the container with id 7a20065f0a641e8ba694230c022967ec35385b49ad3997e9550a1fe3a3c7cfd5 Nov 02 08:20:52.648073 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:52.647976 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" event={"ID":"99ca3f51-620a-47c2-ab1c-4146bc3e08fa","Type":"ContainerStarted","Data":"7a20065f0a641e8ba694230c022967ec35385b49ad3997e9550a1fe3a3c7cfd5"} Nov 02 08:20:53.192207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.192174 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:20:53.198846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.198810 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:20:53.656382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.656343 5055 generic.go:356] "Generic (PLEG): container finished" podID="99ca3f51-620a-47c2-ab1c-4146bc3e08fa" containerID="69e9225cf40f664b1ffd60bc3d88078cc20de87ca4e9a3b3d78bec7a4795b601" exitCode=0 Nov 02 08:20:53.656589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.656432 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" event={"ID":"99ca3f51-620a-47c2-ab1c-4146bc3e08fa","Type":"ContainerDied","Data":"69e9225cf40f664b1ffd60bc3d88078cc20de87ca4e9a3b3d78bec7a4795b601"} Nov 02 08:20:53.753303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.753218 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4"] Nov 02 08:20:53.756322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.756303 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.770288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.770224 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4"] Nov 02 08:20:53.776518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.776473 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.776690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.776629 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9vb\" (UniqueName: \"kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.776868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.776846 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.878130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.878090 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.878363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.878167 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.878363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.878216 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9vb\" (UniqueName: \"kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.878813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.878785 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.880702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.880672 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:53.932377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:53.932340 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9vb\" (UniqueName: \"kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb\") pod \"maas-api-revoke-sub-2a889d4a-9hhh4\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:54.072664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.072576 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:54.449471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.449422 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4"] Nov 02 08:20:54.452209 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:20:54.452179 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f10fe7c_5fb1_4b4e_a57e_061b48a676fb.slice/crio-6d7131c3cbd3b573723caa6bd0278dcba5252ef9f9711581ccada9479caac1ea WatchSource:0}: Error finding container 6d7131c3cbd3b573723caa6bd0278dcba5252ef9f9711581ccada9479caac1ea: Status 404 returned error can't find the container with id 6d7131c3cbd3b573723caa6bd0278dcba5252ef9f9711581ccada9479caac1ea Nov 02 08:20:54.663277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.663237 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" event={"ID":"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb","Type":"ContainerStarted","Data":"6d7131c3cbd3b573723caa6bd0278dcba5252ef9f9711581ccada9479caac1ea"} Nov 02 08:20:54.812696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.812660 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:54.889640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.889612 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg\" (UniqueName: \"kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg\") pod \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " Nov 02 08:20:54.889811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.889707 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca\") pod \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " Nov 02 08:20:54.889811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.889728 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca\") pod \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\" (UID: \"99ca3f51-620a-47c2-ab1c-4146bc3e08fa\") " Nov 02 08:20:54.890180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.890119 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca" pod "99ca3f51-620a-47c2-ab1c-4146bc3e08fa" (UID: "99ca3f51-620a-47c2-ab1c-4146bc3e08fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:20:54.890358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.890316 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:54.892022 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.891992 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg" pod "99ca3f51-620a-47c2-ab1c-4146bc3e08fa" (UID: "99ca3f51-620a-47c2-ab1c-4146bc3e08fa"). InnerVolumeSpecName "kube-api-access-skgzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:20:54.892633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.892275 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca" pod "99ca3f51-620a-47c2-ab1c-4146bc3e08fa" (UID: "99ca3f51-620a-47c2-ab1c-4146bc3e08fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:20:54.920585 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:20:54.920522 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:20:54.991254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.991128 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:54.991254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:54.991206 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skgzg\" (UniqueName: \"kubernetes.io/projected/99ca3f51-620a-47c2-ab1c-4146bc3e08fa-kube-api-access-skgzg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:55.671117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:55.671018 5055 generic.go:356] "Generic (PLEG): container finished" podID="8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" containerID="fe50d9ab80ac99eabf97004e59a0e3a4b5963470a219c37e195271ef93f1cf88" exitCode=0 Nov 02 08:20:55.671552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:55.671110 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" event={"ID":"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb","Type":"ContainerDied","Data":"fe50d9ab80ac99eabf97004e59a0e3a4b5963470a219c37e195271ef93f1cf88"} Nov 02 08:20:55.672701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:55.672672 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" event={"ID":"99ca3f51-620a-47c2-ab1c-4146bc3e08fa","Type":"ContainerDied","Data":"7a20065f0a641e8ba694230c022967ec35385b49ad3997e9550a1fe3a3c7cfd5"} Nov 02 08:20:55.672841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:55.672716 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a20065f0a641e8ba694230c022967ec35385b49ad3997e9550a1fe3a3c7cfd5" Nov 02 08:20:55.672841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:55.672744 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4" Nov 02 08:20:56.837668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.837639 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:56.912096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.912060 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb\" (UniqueName: \"kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb\") pod \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " Nov 02 08:20:56.912299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.912192 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca\") pod \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " Nov 02 08:20:56.912390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.912316 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca\") pod \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\" (UID: \"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb\") " Nov 02 08:20:56.912550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.912524 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca" pod "8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" (UID: "8f10fe7c-5fb1-4b4e-a57e-061b48a676fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:20:56.912700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.912682 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:56.914292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.914267 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb" pod "8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" (UID: "8f10fe7c-5fb1-4b4e-a57e-061b48a676fb"). InnerVolumeSpecName "kube-api-access-gd9vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:20:56.914381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:56.914296 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca" pod "8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" (UID: "8f10fe7c-5fb1-4b4e-a57e-061b48a676fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:20:57.013393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:57.013356 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:57.013393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:57.013393 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9vb\" (UniqueName: \"kubernetes.io/projected/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb-kube-api-access-gd9vb\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:20:57.686979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:57.686944 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" event={"ID":"8f10fe7c-5fb1-4b4e-a57e-061b48a676fb","Type":"ContainerDied","Data":"6d7131c3cbd3b573723caa6bd0278dcba5252ef9f9711581ccada9479caac1ea"} Nov 02 08:20:57.687190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:57.686989 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4" Nov 02 08:20:57.687190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:20:57.686998 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7131c3cbd3b573723caa6bd0278dcba5252ef9f9711581ccada9479caac1ea" Nov 02 08:21:03.911876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:03.911832 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:21:11.670064 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.670024 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4"] Nov 02 08:21:11.670816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.670792 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" containerName="revoke-keys" Nov 02 08:21:11.670909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.670817 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" containerName="revoke-keys" Nov 02 08:21:11.670909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.670861 5055 cpu_manager.go:414] "containerMap: removing container" podUID="99ca3f51-620a-47c2-ab1c-4146bc3e08fa" containerName="revoke-keys" Nov 02 08:21:11.670909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.670870 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ca3f51-620a-47c2-ab1c-4146bc3e08fa" containerName="revoke-keys" Nov 02 08:21:11.671060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.671019 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" containerName="revoke-keys" Nov 02 08:21:11.671060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.671040 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ca3f51-620a-47c2-ab1c-4146bc3e08fa" containerName="revoke-keys" Nov 02 08:21:11.673719 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.673696 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.724103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.724064 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:21:11.769853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.769806 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:21:11.769853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.769817 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:21:11.769853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.769847 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:21:11.772766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.772743 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4"] Nov 02 08:21:11.777623 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.777602 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt"] Nov 02 08:21:11.781291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.780691 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.802686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.802659 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.802859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.802727 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzm9\" (UniqueName: \"kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.802859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.802824 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.832397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.832348 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"premium-w2-10d2fb-kserve-self-signed-certs\"" Nov 02 08:21:11.835573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.835539 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt"] Nov 02 08:21:11.904526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904489 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.904742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904563 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.904742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904648 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.904742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904724 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.904905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904755 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.904905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904866 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzm9\" (UniqueName: \"kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.904905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904896 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.905060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.904935 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.905114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.905085 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s54h\" (UniqueName: \"kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:11.905404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.905386 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.907564 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.907540 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.957620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.957595 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzm9\" (UniqueName: \"kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9\") pod \"maas-api-revoke-sub-d57c420b-g28n4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:11.987732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:11.987688 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:12.005933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.005884 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006062 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s54h\" (UniqueName: \"kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006389 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006364 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006435 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006630 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006664 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006688 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.006863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006693 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.007298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.006908 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.009670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.009647 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.010533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.010513 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.061311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.061020 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s54h\" (UniqueName: \"kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h\") pod \"premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.092217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.092173 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:12.358092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.358040 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4"] Nov 02 08:21:12.359558 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:21:12.359533 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae8589d_07ad_4186_8353_90b9a8dd25f4.slice/crio-0fd4dd1dd891fd8df2923797f626af7bdb1d276d43fe91e87bf88e46222be2cc WatchSource:0}: Error finding container 0fd4dd1dd891fd8df2923797f626af7bdb1d276d43fe91e87bf88e46222be2cc: Status 404 returned error can't find the container with id 0fd4dd1dd891fd8df2923797f626af7bdb1d276d43fe91e87bf88e46222be2cc Nov 02 08:21:12.485392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.485337 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt"] Nov 02 08:21:12.485879 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:21:12.485842 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32bad2f0_4316_4b3e_9d90_9748e059792d.slice/crio-174329f2ccb42bd23e3527c1adb9577a65d68856e7584a1eb1493935f040c33a WatchSource:0}: Error finding container 174329f2ccb42bd23e3527c1adb9577a65d68856e7584a1eb1493935f040c33a: Status 404 returned error can't find the container with id 174329f2ccb42bd23e3527c1adb9577a65d68856e7584a1eb1493935f040c33a Nov 02 08:21:12.792778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.792727 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" event={"ID":"7ae8589d-07ad-4186-8353-90b9a8dd25f4","Type":"ContainerStarted","Data":"d13a621be2589f53809cac8c95abbf8e165a54823c99abc62c0eee12dc03c8c9"} Nov 02 08:21:12.793451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.792801 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" event={"ID":"7ae8589d-07ad-4186-8353-90b9a8dd25f4","Type":"ContainerStarted","Data":"0fd4dd1dd891fd8df2923797f626af7bdb1d276d43fe91e87bf88e46222be2cc"} Nov 02 08:21:12.795299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.795256 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" event={"ID":"32bad2f0-4316-4b3e-9d90-9748e059792d","Type":"ContainerStarted","Data":"fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb"} Nov 02 08:21:12.795649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.795317 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" event={"ID":"32bad2f0-4316-4b3e-9d90-9748e059792d","Type":"ContainerStarted","Data":"174329f2ccb42bd23e3527c1adb9577a65d68856e7584a1eb1493935f040c33a"} Nov 02 08:21:12.898418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:12.898364 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" podStartSLOduration=1.898345929 podStartE2EDuration="1.898345929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:12.896986364 +0000 UTC m=+919.987606777" watchObservedRunningTime="2026-11-02 08:21:12.898345929 +0000 UTC m=+919.988966352" Nov 02 08:21:13.004617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:13.004545 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podStartSLOduration=2.004524768 podStartE2EDuration="2.004524768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:13.002423238 +0000 UTC m=+920.093043659" watchObservedRunningTime="2026-11-02 08:21:13.004524768 +0000 UTC m=+920.095145190" Nov 02 08:21:13.095484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:13.095448 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:13.821322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:13.821275 5055 generic.go:356] "Generic (PLEG): container finished" podID="7ae8589d-07ad-4186-8353-90b9a8dd25f4" containerID="d13a621be2589f53809cac8c95abbf8e165a54823c99abc62c0eee12dc03c8c9" exitCode=0 Nov 02 08:21:13.822544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:13.822509 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" event={"ID":"7ae8589d-07ad-4186-8353-90b9a8dd25f4","Type":"ContainerDied","Data":"d13a621be2589f53809cac8c95abbf8e165a54823c99abc62c0eee12dc03c8c9"} Nov 02 08:21:14.981565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:14.981538 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:15.061786 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.061749 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca\") pod \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " Nov 02 08:21:15.061786 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.061802 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9\" (UniqueName: \"kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9\") pod \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " Nov 02 08:21:15.062074 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.061930 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca\") pod \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\" (UID: \"7ae8589d-07ad-4186-8353-90b9a8dd25f4\") " Nov 02 08:21:15.062175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.062126 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca" pod "7ae8589d-07ad-4186-8353-90b9a8dd25f4" (UID: "7ae8589d-07ad-4186-8353-90b9a8dd25f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:21:15.062368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.062349 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:15.064089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.064055 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca" pod "7ae8589d-07ad-4186-8353-90b9a8dd25f4" (UID: "7ae8589d-07ad-4186-8353-90b9a8dd25f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:21:15.064209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.064114 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9" pod "7ae8589d-07ad-4186-8353-90b9a8dd25f4" (UID: "7ae8589d-07ad-4186-8353-90b9a8dd25f4"). InnerVolumeSpecName "kube-api-access-dzzm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:15.163024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.162990 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzzm9\" (UniqueName: \"kubernetes.io/projected/7ae8589d-07ad-4186-8353-90b9a8dd25f4-kube-api-access-dzzm9\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:15.163024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.163020 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae8589d-07ad-4186-8353-90b9a8dd25f4-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:15.835251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.835214 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" event={"ID":"7ae8589d-07ad-4186-8353-90b9a8dd25f4","Type":"ContainerDied","Data":"0fd4dd1dd891fd8df2923797f626af7bdb1d276d43fe91e87bf88e46222be2cc"} Nov 02 08:21:15.835251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.835232 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4" Nov 02 08:21:15.835440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:15.835269 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd4dd1dd891fd8df2923797f626af7bdb1d276d43fe91e87bf88e46222be2cc" Nov 02 08:21:19.208369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.208330 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7"] Nov 02 08:21:19.208859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.208844 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7ae8589d-07ad-4186-8353-90b9a8dd25f4" containerName="revoke-keys" Nov 02 08:21:19.208907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.208859 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae8589d-07ad-4186-8353-90b9a8dd25f4" containerName="revoke-keys" Nov 02 08:21:19.208942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.208932 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae8589d-07ad-4186-8353-90b9a8dd25f4" containerName="revoke-keys" Nov 02 08:21:19.211545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.211523 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.267392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.267353 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7"] Nov 02 08:21:19.308416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.308382 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:21:19.308416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.308395 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:21:19.308650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.308429 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:21:19.308650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.308523 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:21:19.313486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.313458 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.313640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.313622 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.313695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.313647 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9sn\" (UniqueName: \"kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.415106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.415072 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.415331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.415242 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.415331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.415270 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9sn\" (UniqueName: \"kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.415905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.415877 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.417645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.417616 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.468835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.468760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9sn\" (UniqueName: \"kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn\") pod \"maas-api-revoke-sub-d50bc390-ms5f7\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.525200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.525165 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:19.904561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:19.904530 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7"] Nov 02 08:21:19.905383 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:21:19.905334 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184d9e8a_6044_454a_b583_9ab4cf644045.slice/crio-ab8d68c9f8f8a10e91add57f0b5fa252adce01f0a0247adce06f21a6295b0c69 WatchSource:0}: Error finding container ab8d68c9f8f8a10e91add57f0b5fa252adce01f0a0247adce06f21a6295b0c69: Status 404 returned error can't find the container with id ab8d68c9f8f8a10e91add57f0b5fa252adce01f0a0247adce06f21a6295b0c69 Nov 02 08:21:20.889247 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:20.889208 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" event={"ID":"184d9e8a-6044-454a-b583-9ab4cf644045","Type":"ContainerStarted","Data":"d3a50c87a9fc84f0f3b69ddda579e2e03f63260d5a4cf230aed94b5f751742c6"} Nov 02 08:21:20.889632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:20.889277 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" event={"ID":"184d9e8a-6044-454a-b583-9ab4cf644045","Type":"ContainerStarted","Data":"ab8d68c9f8f8a10e91add57f0b5fa252adce01f0a0247adce06f21a6295b0c69"} Nov 02 08:21:20.995609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:20.995561 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" podStartSLOduration=1.995544628 podStartE2EDuration="1.995544628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:20.994332853 +0000 UTC m=+928.084953274" watchObservedRunningTime="2026-11-02 08:21:20.995544628 +0000 UTC m=+928.086165047" Nov 02 08:21:21.897068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:21.897033 5055 generic.go:356] "Generic (PLEG): container finished" podID="184d9e8a-6044-454a-b583-9ab4cf644045" containerID="d3a50c87a9fc84f0f3b69ddda579e2e03f63260d5a4cf230aed94b5f751742c6" exitCode=0 Nov 02 08:21:21.897540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:21.897115 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" event={"ID":"184d9e8a-6044-454a-b583-9ab4cf644045","Type":"ContainerDied","Data":"d3a50c87a9fc84f0f3b69ddda579e2e03f63260d5a4cf230aed94b5f751742c6"} Nov 02 08:21:22.093315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.093282 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:22.105739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.105711 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:22.413702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.413665 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj"] Nov 02 08:21:22.416782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.416759 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.473456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.473419 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj"] Nov 02 08:21:22.554424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.554386 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.554623 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.554592 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.554742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.554727 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.655657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.655621 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.655848 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.655692 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.655848 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.655828 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.656606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.656566 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.658350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.658321 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.708560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.708530 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf\") pod \"maas-api-revoke-sub-620f49b9-tmxbj\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.730335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.730303 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:22.917189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:22.917122 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:21:23.070389 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.070348 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:23.129304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.129266 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj"] Nov 02 08:21:23.131750 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:21:23.131703 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270e2570_d15e_4ae7_9eed_d64e90ee5407.slice/crio-a2200e8a5748725c7bcea8a82376e984c44522e90678787f0640d4846133bc90 WatchSource:0}: Error finding container a2200e8a5748725c7bcea8a82376e984c44522e90678787f0640d4846133bc90: Status 404 returned error can't find the container with id a2200e8a5748725c7bcea8a82376e984c44522e90678787f0640d4846133bc90 Nov 02 08:21:23.161904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.161863 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca\") pod \"184d9e8a-6044-454a-b583-9ab4cf644045\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " Nov 02 08:21:23.162225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.161919 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca\") pod \"184d9e8a-6044-454a-b583-9ab4cf644045\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " Nov 02 08:21:23.162225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.161946 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn\" (UniqueName: \"kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn\") pod \"184d9e8a-6044-454a-b583-9ab4cf644045\" (UID: \"184d9e8a-6044-454a-b583-9ab4cf644045\") " Nov 02 08:21:23.162428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.162397 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca" pod "184d9e8a-6044-454a-b583-9ab4cf644045" (UID: "184d9e8a-6044-454a-b583-9ab4cf644045"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:21:23.162620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.162586 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:23.164403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.164372 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn" pod "184d9e8a-6044-454a-b583-9ab4cf644045" (UID: "184d9e8a-6044-454a-b583-9ab4cf644045"). InnerVolumeSpecName "kube-api-access-wk9sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:23.164553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.164475 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca" pod "184d9e8a-6044-454a-b583-9ab4cf644045" (UID: "184d9e8a-6044-454a-b583-9ab4cf644045"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:21:23.263728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.263643 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184d9e8a-6044-454a-b583-9ab4cf644045-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:23.263728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.263679 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9sn\" (UniqueName: \"kubernetes.io/projected/184d9e8a-6044-454a-b583-9ab4cf644045-kube-api-access-wk9sn\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:23.911771 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.911736 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" Nov 02 08:21:23.911975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.911769 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7" event={"ID":"184d9e8a-6044-454a-b583-9ab4cf644045","Type":"ContainerDied","Data":"ab8d68c9f8f8a10e91add57f0b5fa252adce01f0a0247adce06f21a6295b0c69"} Nov 02 08:21:23.911975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.911826 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8d68c9f8f8a10e91add57f0b5fa252adce01f0a0247adce06f21a6295b0c69" Nov 02 08:21:23.913644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.913613 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" event={"ID":"270e2570-d15e-4ae7-9eed-d64e90ee5407","Type":"ContainerStarted","Data":"2ddba2846ce0843099c03b5595beb72401b536a03c885f91201ac58b6a2462fb"} Nov 02 08:21:23.913803 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:23.913673 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" event={"ID":"270e2570-d15e-4ae7-9eed-d64e90ee5407","Type":"ContainerStarted","Data":"a2200e8a5748725c7bcea8a82376e984c44522e90678787f0640d4846133bc90"} Nov 02 08:21:24.023741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:24.023683 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" podStartSLOduration=2.023668237 podStartE2EDuration="2.023668237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:24.021304613 +0000 UTC m=+931.111925033" watchObservedRunningTime="2026-11-02 08:21:24.023668237 +0000 UTC m=+931.114288655" Nov 02 08:21:24.923104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:24.923069 5055 generic.go:356] "Generic (PLEG): container finished" podID="270e2570-d15e-4ae7-9eed-d64e90ee5407" containerID="2ddba2846ce0843099c03b5595beb72401b536a03c885f91201ac58b6a2462fb" exitCode=0 Nov 02 08:21:24.923322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:24.923106 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" event={"ID":"270e2570-d15e-4ae7-9eed-d64e90ee5407","Type":"ContainerDied","Data":"2ddba2846ce0843099c03b5595beb72401b536a03c885f91201ac58b6a2462fb"} Nov 02 08:21:24.924838 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:21:24.924802 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:21:26.077879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.077856 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:26.199071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.199044 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca\") pod \"270e2570-d15e-4ae7-9eed-d64e90ee5407\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " Nov 02 08:21:26.199279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.199090 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca\") pod \"270e2570-d15e-4ae7-9eed-d64e90ee5407\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " Nov 02 08:21:26.199359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.199307 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf\") pod \"270e2570-d15e-4ae7-9eed-d64e90ee5407\" (UID: \"270e2570-d15e-4ae7-9eed-d64e90ee5407\") " Nov 02 08:21:26.199488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.199461 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca" pod "270e2570-d15e-4ae7-9eed-d64e90ee5407" (UID: "270e2570-d15e-4ae7-9eed-d64e90ee5407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:21:26.199852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.199829 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:26.201498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.201466 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf" pod "270e2570-d15e-4ae7-9eed-d64e90ee5407" (UID: "270e2570-d15e-4ae7-9eed-d64e90ee5407"). InnerVolumeSpecName "kube-api-access-4bkvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:26.201629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.201605 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca" pod "270e2570-d15e-4ae7-9eed-d64e90ee5407" (UID: "270e2570-d15e-4ae7-9eed-d64e90ee5407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:21:26.300733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.300693 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/270e2570-d15e-4ae7-9eed-d64e90ee5407-kube-api-access-4bkvf\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:26.300733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.300723 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270e2570-d15e-4ae7-9eed-d64e90ee5407-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:26.941944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.941904 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" event={"ID":"270e2570-d15e-4ae7-9eed-d64e90ee5407","Type":"ContainerDied","Data":"a2200e8a5748725c7bcea8a82376e984c44522e90678787f0640d4846133bc90"} Nov 02 08:21:26.941944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.941939 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj" Nov 02 08:21:26.942248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:26.941982 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2200e8a5748725c7bcea8a82376e984c44522e90678787f0640d4846133bc90" Nov 02 08:21:29.317626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:29.317593 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:21:54.923850 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:21:54.923809 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:21:56.843996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:21:56.843959 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:22:01.244758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.244721 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn"] Nov 02 08:22:01.245452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.245430 5055 cpu_manager.go:414] "containerMap: removing container" podUID="184d9e8a-6044-454a-b583-9ab4cf644045" containerName="revoke-keys" Nov 02 08:22:01.245550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.245456 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184d9e8a-6044-454a-b583-9ab4cf644045" containerName="revoke-keys" Nov 02 08:22:01.245550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.245500 5055 cpu_manager.go:414] "containerMap: removing container" podUID="270e2570-d15e-4ae7-9eed-d64e90ee5407" containerName="revoke-keys" Nov 02 08:22:01.245550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.245510 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270e2570-d15e-4ae7-9eed-d64e90ee5407" containerName="revoke-keys" Nov 02 08:22:01.245704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.245673 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="270e2570-d15e-4ae7-9eed-d64e90ee5407" containerName="revoke-keys" Nov 02 08:22:01.245704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.245693 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="184d9e8a-6044-454a-b583-9ab4cf644045" containerName="revoke-keys" Nov 02 08:22:01.249053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.249029 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.336243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.336195 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtprq\" (UniqueName: \"kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.336430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.336339 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.336475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.336437 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.336531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.336515 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.336589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.336578 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.336754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.336730 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.344255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.344222 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-096685\"/\"premium-w6-096685-kserve-self-signed-certs\"" Nov 02 08:22:01.347001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.346974 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn"] Nov 02 08:22:01.437905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.437865 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.437918 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtprq\" (UniqueName: \"kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.437964 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.438025 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.438104 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.438176 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.438430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.438619 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.438762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.438740 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.440412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.440384 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.440775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.440753 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.494655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.494615 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtprq\" (UniqueName: \"kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq\") pod \"premium-w6-096685-kserve-fc57fb556-g58zn\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.561329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.561240 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:01.937211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:01.937184 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn"] Nov 02 08:22:01.938827 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:22:01.938798 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda439f420_80fa_4c2d_9a5b_3e2f733233f6.slice/crio-3679827a03b54b2756eb4f2b9784f5e9d87703d518adbc81be2f2f36774f6c46 WatchSource:0}: Error finding container 3679827a03b54b2756eb4f2b9784f5e9d87703d518adbc81be2f2f36774f6c46: Status 404 returned error can't find the container with id 3679827a03b54b2756eb4f2b9784f5e9d87703d518adbc81be2f2f36774f6c46 Nov 02 08:22:02.194747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:02.194658 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" event={"ID":"a439f420-80fa-4c2d-9a5b-3e2f733233f6","Type":"ContainerStarted","Data":"07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8"} Nov 02 08:22:02.194747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:02.194709 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" event={"ID":"a439f420-80fa-4c2d-9a5b-3e2f733233f6","Type":"ContainerStarted","Data":"3679827a03b54b2756eb4f2b9784f5e9d87703d518adbc81be2f2f36774f6c46"} Nov 02 08:22:02.301852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:02.301795 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podStartSLOduration=1.301779059 podStartE2EDuration="1.301779059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:22:02.29998656 +0000 UTC m=+969.390606980" watchObservedRunningTime="2026-11-02 08:22:02.301779059 +0000 UTC m=+969.392399477" Nov 02 08:22:02.561340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:02.561309 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:11.562119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:11.562080 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:11.574949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:11.574920 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:12.301407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:12.301376 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:22:24.921464 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:22:24.921420 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:22:31.761287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:31.761242 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:22:54.921225 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:22:54.921182 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:22:55.239946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.239913 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl"] Nov 02 08:22:55.244067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.244044 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.297999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.297968 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl"] Nov 02 08:22:55.322943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.322914 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.323159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.322958 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.323159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.322980 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.323159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.323056 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.323386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.323365 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.323453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.323419 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nps\" (UniqueName: \"kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.340110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.340085 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e6ca80\"/\"kube-root-ca.crt\"" Nov 02 08:22:55.340287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.340217 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e6ca80\"/\"openshift-service-ca.crt\"" Nov 02 08:22:55.340340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.340311 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e6ca80\"/\"distinct-w0-e6ca80-kserve-self-signed-certs\"" Nov 02 08:22:55.340430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.340409 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e6ca80\"/\"default-dockercfg-57rx9\"" Nov 02 08:22:55.425132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425096 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425179 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425213 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425240 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425412 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425457 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nps\" (UniqueName: \"kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425473 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.425698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.425525 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.426055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.426024 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.428254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.428229 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.428498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.428479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.480015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.479975 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nps\" (UniqueName: \"kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps\") pod \"distinct-w0-e6ca80-kserve-769b7745c4-dswfl\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.556585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.556484 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:22:55.958525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:55.958500 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl"] Nov 02 08:22:55.959999 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:22:55.959967 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd33b3ab9_b4bc_4f43_af95_313f6e8cda4f.slice/crio-54944e946cf4eee396f150a8b9811f55182a82d8cf77f1fa7de664eb85bd2080 WatchSource:0}: Error finding container 54944e946cf4eee396f150a8b9811f55182a82d8cf77f1fa7de664eb85bd2080: Status 404 returned error can't find the container with id 54944e946cf4eee396f150a8b9811f55182a82d8cf77f1fa7de664eb85bd2080 Nov 02 08:22:56.632339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:56.632296 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" event={"ID":"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f","Type":"ContainerStarted","Data":"0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3"} Nov 02 08:22:56.632541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:56.632354 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" event={"ID":"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f","Type":"ContainerStarted","Data":"54944e946cf4eee396f150a8b9811f55182a82d8cf77f1fa7de664eb85bd2080"} Nov 02 08:22:56.745413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:56.745360 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podStartSLOduration=1.7453449509999999 podStartE2EDuration="1.745344951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:22:56.743769512 +0000 UTC m=+1023.834389930" watchObservedRunningTime="2026-11-02 08:22:56.745344951 +0000 UTC m=+1023.835965371" Nov 02 08:22:57.557625 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:22:57.557590 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:23:02.632784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:02.632731 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:23:05.408528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.408442 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw"] Nov 02 08:23:05.413400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.413366 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.508570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.508531 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"openshift-service-ca.crt\"" Nov 02 08:23:05.508840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.508638 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"default-dockercfg-zz4bc\"" Nov 02 08:23:05.508840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.508719 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"premium-w4-8e6407-kserve-self-signed-certs\"" Nov 02 08:23:05.508840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.508726 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"kube-root-ca.crt\"" Nov 02 08:23:05.512042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.512014 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw"] Nov 02 08:23:05.547943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.547905 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.548134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.547967 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.548134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.548076 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.548134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.548104 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knssp\" (UniqueName: \"kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.548346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.548160 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.548346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.548306 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.557524 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.557490 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:23:05.588476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.588438 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:23:05.649756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.649723 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.649756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.649762 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.649790 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.649814 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knssp\" (UniqueName: \"kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.649834 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.649879 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650271 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.650249 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.650340 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.650509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.650489 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.652232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.652211 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.652455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.652439 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.704076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.704035 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knssp\" (UniqueName: \"kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp\") pod \"premium-w4-8e6407-kserve-6b58695b6-57fnw\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:05.704269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.704219 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:23:05.729641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:05.729606 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:06.143395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:06.143370 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw"] Nov 02 08:23:06.144744 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:23:06.144706 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9af39d1_64a6_4c73_a63b_32b614bba48f.slice/crio-9842cc6f253e1fd2ca8c06f0531b0d282acec5b892199e6242d953f58aac37b2 WatchSource:0}: Error finding container 9842cc6f253e1fd2ca8c06f0531b0d282acec5b892199e6242d953f58aac37b2: Status 404 returned error can't find the container with id 9842cc6f253e1fd2ca8c06f0531b0d282acec5b892199e6242d953f58aac37b2 Nov 02 08:23:06.701929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:06.701892 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" event={"ID":"c9af39d1-64a6-4c73-a63b-32b614bba48f","Type":"ContainerStarted","Data":"68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b"} Nov 02 08:23:06.702454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:06.701959 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" event={"ID":"c9af39d1-64a6-4c73-a63b-32b614bba48f","Type":"ContainerStarted","Data":"9842cc6f253e1fd2ca8c06f0531b0d282acec5b892199e6242d953f58aac37b2"} Nov 02 08:23:06.729702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:06.729674 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:06.810008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:06.809952 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podStartSLOduration=1.8099390340000001 podStartE2EDuration="1.809939034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:23:06.807960835 +0000 UTC m=+1033.898581253" watchObservedRunningTime="2026-11-02 08:23:06.809939034 +0000 UTC m=+1033.900559452" Nov 02 08:23:14.175725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.175673 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk"] Nov 02 08:23:14.179704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.179684 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.274949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.274900 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:23:14.274949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.274931 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:23:14.274949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.274953 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:23:14.275332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.275019 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:23:14.279214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.279182 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk"] Nov 02 08:23:14.360373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.360332 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhnt\" (UniqueName: \"kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.360373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.360377 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.360600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.360466 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.461461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.461419 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhnt\" (UniqueName: \"kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.461679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.461470 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.461679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.461503 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.462187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.462168 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.463877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.463854 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.514477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.514440 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhnt\" (UniqueName: \"kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt\") pod \"maas-api-revoke-sub-606d263c-zj7rk\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:14.794189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:14.794086 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:15.163650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.163606 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk"] Nov 02 08:23:15.165133 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:23:15.165103 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39837395_1069_418c_a113_142d1cd57901.slice/crio-a736086c2aaf4c4ec33e2f65042ddb901b13f275939b626709609f0800a7c698 WatchSource:0}: Error finding container a736086c2aaf4c4ec33e2f65042ddb901b13f275939b626709609f0800a7c698: Status 404 returned error can't find the container with id a736086c2aaf4c4ec33e2f65042ddb901b13f275939b626709609f0800a7c698 Nov 02 08:23:15.730010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.729968 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:15.742936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.742854 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:15.776013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.775982 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" event={"ID":"39837395-1069-418c-a113-142d1cd57901","Type":"ContainerStarted","Data":"3c2d7781ff3c81beab9b80c1c66650aa575eafea21e13b633e770f14c9ba01c0"} Nov 02 08:23:15.776224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.776038 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" event={"ID":"39837395-1069-418c-a113-142d1cd57901","Type":"ContainerStarted","Data":"a736086c2aaf4c4ec33e2f65042ddb901b13f275939b626709609f0800a7c698"} Nov 02 08:23:15.787804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.787777 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:23:15.958491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:15.958433 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" podStartSLOduration=1.958420659 podStartE2EDuration="1.958420659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:23:15.955907754 +0000 UTC m=+1043.046528174" watchObservedRunningTime="2026-11-02 08:23:15.958420659 +0000 UTC m=+1043.049041113" Nov 02 08:23:16.784790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:16.784751 5055 generic.go:356] "Generic (PLEG): container finished" podID="39837395-1069-418c-a113-142d1cd57901" containerID="3c2d7781ff3c81beab9b80c1c66650aa575eafea21e13b633e770f14c9ba01c0" exitCode=0 Nov 02 08:23:16.785278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:16.784830 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" event={"ID":"39837395-1069-418c-a113-142d1cd57901","Type":"ContainerDied","Data":"3c2d7781ff3c81beab9b80c1c66650aa575eafea21e13b633e770f14c9ba01c0"} Nov 02 08:23:17.964704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:17.964679 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:18.007024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.006987 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt\" (UniqueName: \"kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt\") pod \"39837395-1069-418c-a113-142d1cd57901\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " Nov 02 08:23:18.007223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.007036 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca\") pod \"39837395-1069-418c-a113-142d1cd57901\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " Nov 02 08:23:18.007223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.007193 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca\") pod \"39837395-1069-418c-a113-142d1cd57901\" (UID: \"39837395-1069-418c-a113-142d1cd57901\") " Nov 02 08:23:18.007439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.007411 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca" pod "39837395-1069-418c-a113-142d1cd57901" (UID: "39837395-1069-418c-a113-142d1cd57901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:23:18.007795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.007747 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39837395-1069-418c-a113-142d1cd57901-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:23:18.009393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.009363 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca" pod "39837395-1069-418c-a113-142d1cd57901" (UID: "39837395-1069-418c-a113-142d1cd57901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:23:18.009481 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.009435 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt" pod "39837395-1069-418c-a113-142d1cd57901" (UID: "39837395-1069-418c-a113-142d1cd57901"). InnerVolumeSpecName "kube-api-access-mnhnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:23:18.108870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.108769 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhnt\" (UniqueName: \"kubernetes.io/projected/39837395-1069-418c-a113-142d1cd57901-kube-api-access-mnhnt\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:23:18.108870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.108812 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39837395-1069-418c-a113-142d1cd57901-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:23:18.806010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.805973 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" Nov 02 08:23:18.806280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.805972 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk" event={"ID":"39837395-1069-418c-a113-142d1cd57901","Type":"ContainerDied","Data":"a736086c2aaf4c4ec33e2f65042ddb901b13f275939b626709609f0800a7c698"} Nov 02 08:23:18.806280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:18.806254 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a736086c2aaf4c4ec33e2f65042ddb901b13f275939b626709609f0800a7c698" Nov 02 08:23:24.922439 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:23:24.922402 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:23:35.446797 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:35.446761 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:23:53.263294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:53.263263 5055 scope.go:122] "RemoveContainer" containerID="2c464705c858af0337442c5ca43dfe1c58b0d83b68a1337b10d1668c9ce14b49" Nov 02 08:23:54.924616 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:23:54.924572 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:23:58.856961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.856911 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq"] Nov 02 08:23:58.858292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.858257 5055 cpu_manager.go:414] "containerMap: removing container" podUID="39837395-1069-418c-a113-142d1cd57901" containerName="revoke-keys" Nov 02 08:23:58.858292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.858291 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39837395-1069-418c-a113-142d1cd57901" containerName="revoke-keys" Nov 02 08:23:58.858617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.858596 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="39837395-1069-418c-a113-142d1cd57901" containerName="revoke-keys" Nov 02 08:23:58.865834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.865803 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.915791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.915752 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq"] Nov 02 08:23:58.933677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.933638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.933677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.933682 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.933967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.933762 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6k7\" (UniqueName: \"kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.933967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.933887 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.934159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.933965 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.934234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.934154 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:58.957497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:58.957458 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-096685\"/\"distinct-w6-096685-kserve-self-signed-certs\"" Nov 02 08:23:59.034975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.034928 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.034975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.034975 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035003 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6k7\" (UniqueName: \"kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035053 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035072 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035160 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035484 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035510 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.035638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.035564 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.037636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.037614 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.037915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.037898 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.089113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.089081 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6k7\" (UniqueName: \"kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7\") pod \"distinct-w6-096685-kserve-9bbc79bc-jmzqq\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.179066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.178975 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:23:59.599829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:23:59.595513 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq"] Nov 02 08:24:00.120401 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:00.120363 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" event={"ID":"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609","Type":"ContainerStarted","Data":"11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf"} Nov 02 08:24:00.120815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:00.120416 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" event={"ID":"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609","Type":"ContainerStarted","Data":"db9ad8ea45821dfdbe958105c0c4e03466f5edba8b30782dbb5b0558b3a6e736"} Nov 02 08:24:00.179994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:00.179958 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:24:00.229117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:00.229059 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podStartSLOduration=2.229044853 podStartE2EDuration="2.229044853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:00.225686923 +0000 UTC m=+1087.316307343" watchObservedRunningTime="2026-11-02 08:24:00.229044853 +0000 UTC m=+1087.319665272" Nov 02 08:24:00.423208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:00.423112 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:24:07.166236 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.166202 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd"] Nov 02 08:24:07.177431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.177400 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.224435 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.224399 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd"] Nov 02 08:24:07.265445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.265410 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:24:07.265445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.265432 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:24:07.265445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.265444 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:24:07.271392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.271366 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:24:07.346193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.346127 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.346380 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.346261 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.346380 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.346363 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhn7c\" (UniqueName: \"kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.447928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.447834 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.447928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.447917 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.448193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.448009 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhn7c\" (UniqueName: \"kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.448757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.448729 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.450384 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.450362 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.501766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.501723 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhn7c\" (UniqueName: \"kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c\") pod \"maas-api-revoke-sub-d5ccb32d-2pnwd\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:07.792098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:07.792053 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:08.161761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:08.161736 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd"] Nov 02 08:24:08.163530 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:24:08.163493 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd005872_8df4_49e0_9710_ca51e6908619.slice/crio-15635a9bf8d60fc985b59fe31f5ed8f19fa4aa8dc243edb1bb3a1bf2e6c546b5 WatchSource:0}: Error finding container 15635a9bf8d60fc985b59fe31f5ed8f19fa4aa8dc243edb1bb3a1bf2e6c546b5: Status 404 returned error can't find the container with id 15635a9bf8d60fc985b59fe31f5ed8f19fa4aa8dc243edb1bb3a1bf2e6c546b5 Nov 02 08:24:08.186919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:08.186883 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" event={"ID":"cd005872-8df4-49e0-9710-ca51e6908619","Type":"ContainerStarted","Data":"15635a9bf8d60fc985b59fe31f5ed8f19fa4aa8dc243edb1bb3a1bf2e6c546b5"} Nov 02 08:24:08.978633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:08.978590 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59"] Nov 02 08:24:08.978960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:08.978899 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" containerID="cri-o://7ab7198eb76c1004cd558996a7146d09da214e7e9c84a763cd787334be064c3d" gracePeriod=30 Nov 02 08:24:09.179416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:09.179378 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:24:09.192248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:09.192213 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:24:09.203326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:09.203290 5055 generic.go:356] "Generic (PLEG): container finished" podID="cd005872-8df4-49e0-9710-ca51e6908619" containerID="9f9808c419ef650b97cce91098fe39f337cf986cde8c84fa465e9ee800c62348" exitCode=0 Nov 02 08:24:09.203495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:09.203377 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" event={"ID":"cd005872-8df4-49e0-9710-ca51e6908619","Type":"ContainerDied","Data":"9f9808c419ef650b97cce91098fe39f337cf986cde8c84fa465e9ee800c62348"} Nov 02 08:24:09.215602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:09.215572 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:24:10.376023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.375997 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:10.490120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.490016 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c\" (UniqueName: \"kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c\") pod \"cd005872-8df4-49e0-9710-ca51e6908619\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " Nov 02 08:24:10.490307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.490164 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca\") pod \"cd005872-8df4-49e0-9710-ca51e6908619\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " Nov 02 08:24:10.490307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.490207 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca\") pod \"cd005872-8df4-49e0-9710-ca51e6908619\" (UID: \"cd005872-8df4-49e0-9710-ca51e6908619\") " Nov 02 08:24:10.490605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.490573 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca" pod "cd005872-8df4-49e0-9710-ca51e6908619" (UID: "cd005872-8df4-49e0-9710-ca51e6908619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:24:10.490744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.490702 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:10.492733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.492704 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca" pod "cd005872-8df4-49e0-9710-ca51e6908619" (UID: "cd005872-8df4-49e0-9710-ca51e6908619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:24:10.492851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.492817 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c" pod "cd005872-8df4-49e0-9710-ca51e6908619" (UID: "cd005872-8df4-49e0-9710-ca51e6908619"). InnerVolumeSpecName "kube-api-access-jhn7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:24:10.592160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.592118 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhn7c\" (UniqueName: \"kubernetes.io/projected/cd005872-8df4-49e0-9710-ca51e6908619-kube-api-access-jhn7c\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:10.592160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:10.592163 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd005872-8df4-49e0-9710-ca51e6908619-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:11.220299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.220263 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" event={"ID":"cd005872-8df4-49e0-9710-ca51e6908619","Type":"ContainerDied","Data":"15635a9bf8d60fc985b59fe31f5ed8f19fa4aa8dc243edb1bb3a1bf2e6c546b5"} Nov 02 08:24:11.220469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.220311 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15635a9bf8d60fc985b59fe31f5ed8f19fa4aa8dc243edb1bb3a1bf2e6c546b5" Nov 02 08:24:11.220469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.220309 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd" Nov 02 08:24:11.893275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.893231 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm"] Nov 02 08:24:11.893833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.893814 5055 cpu_manager.go:414] "containerMap: removing container" podUID="cd005872-8df4-49e0-9710-ca51e6908619" containerName="revoke-keys" Nov 02 08:24:11.893879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.893834 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd005872-8df4-49e0-9710-ca51e6908619" containerName="revoke-keys" Nov 02 08:24:11.893968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.893958 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd005872-8df4-49e0-9710-ca51e6908619" containerName="revoke-keys" Nov 02 08:24:11.897540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.897517 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:11.946222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.946192 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9d77b6-openshift-default-dockercfg-r279j\"" Nov 02 08:24:11.996568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:11.996525 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm"] Nov 02 08:24:12.008112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008080 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008205 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008314 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008408 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008504 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008541 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008655 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rn9k\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008682 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.008708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.008700 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110074 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110116 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110187 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110209 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rn9k\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110296 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110318 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110369 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110438 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110692 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110756 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110820 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.110947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110879 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.111006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.110950 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.112644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.112619 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.112860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.112839 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.169876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.169795 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.170087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.170063 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rn9k\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.211720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.211686 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:12.641620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.641585 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm"] Nov 02 08:24:12.642569 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:24:12.642541 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510e13a2_4f9f_43f3_8fa7_9d8ddabce7e5.slice/crio-9b9af52bacb5751b4d2e1ef9b3017ac9ebc86c2947d23b306851b332e963187a WatchSource:0}: Error finding container 9b9af52bacb5751b4d2e1ef9b3017ac9ebc86c2947d23b306851b332e963187a: Status 404 returned error can't find the container with id 9b9af52bacb5751b4d2e1ef9b3017ac9ebc86c2947d23b306851b332e963187a Nov 02 08:24:12.645760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.645719 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:24:12.645831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.645806 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:24:12.645876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:12.645846 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:24:13.240560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:13.240517 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" event={"ID":"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5","Type":"ContainerStarted","Data":"c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8"} Nov 02 08:24:13.240985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:13.240589 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" event={"ID":"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5","Type":"ContainerStarted","Data":"9b9af52bacb5751b4d2e1ef9b3017ac9ebc86c2947d23b306851b332e963187a"} Nov 02 08:24:13.356569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:13.356506 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podStartSLOduration=2.356492466 podStartE2EDuration="2.356492466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:13.354421072 +0000 UTC m=+1100.445041481" watchObservedRunningTime="2026-11-02 08:24:13.356492466 +0000 UTC m=+1100.447112884" Nov 02 08:24:14.220708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.220275 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:14.222387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.222356 5055 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:24:14.222547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.222408 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 08:24:14.254119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.254078 5055 generic.go:356] "Generic (PLEG): container finished" podID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerID="7ab7198eb76c1004cd558996a7146d09da214e7e9c84a763cd787334be064c3d" exitCode=0 Nov 02 08:24:14.254629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.254125 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" event={"ID":"3e83a04b-acfb-4583-b44c-a52d18e16588","Type":"ContainerDied","Data":"7ab7198eb76c1004cd558996a7146d09da214e7e9c84a763cd787334be064c3d"} Nov 02 08:24:14.316334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.316305 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:24:14.336597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336563 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336618 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336668 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336751 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336798 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336827 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336865 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.336961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336872 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:24:14.336961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336914 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:24:14.336961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336922 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.337359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.336970 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy\") pod \"3e83a04b-acfb-4583-b44c-a52d18e16588\" (UID: \"3e83a04b-acfb-4583-b44c-a52d18e16588\") " Nov 02 08:24:14.337359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337113 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:24:14.337359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337207 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:24:14.337359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337319 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:24:14.337795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337771 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.337897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337800 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.337897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337815 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.337897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337830 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.337897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.337844 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e83a04b-acfb-4583-b44c-a52d18e16588-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.339526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.339501 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:24:14.339680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.339660 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:24:14.339756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.339692 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "kube-api-access-mqcbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:24:14.339880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.339861 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo" pod "3e83a04b-acfb-4583-b44c-a52d18e16588" (UID: "3e83a04b-acfb-4583-b44c-a52d18e16588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:24:14.439411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.439374 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.439411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.439414 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqcbc\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-kube-api-access-mqcbc\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.439613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.439427 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:14.439613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:14.439437 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e83a04b-acfb-4583-b44c-a52d18e16588-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:24:15.212943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.212903 5055 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:24:15.213112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.212968 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 08:24:15.262877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.262844 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" Nov 02 08:24:15.262877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.262867 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59" event={"ID":"3e83a04b-acfb-4583-b44c-a52d18e16588","Type":"ContainerDied","Data":"76c5a4bbbe00e4f21aa4de6cd52b94c4812ad56f01fc3cc652fa8568dc1c7924"} Nov 02 08:24:15.263427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.262935 5055 scope.go:122] "RemoveContainer" containerID="7ab7198eb76c1004cd558996a7146d09da214e7e9c84a763cd787334be064c3d" Nov 02 08:24:15.463999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.463970 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59"] Nov 02 08:24:15.467647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:15.467620 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-95j59"] Nov 02 08:24:16.212995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:16.212954 5055 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:24:16.213229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:16.213018 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 08:24:17.110338 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:17.110287 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" path="/var/lib/kubelet/pods/3e83a04b-acfb-4583-b44c-a52d18e16588/volumes" Nov 02 08:24:17.212374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:17.212335 5055 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:24:17.212665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:17.212635 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 08:24:18.212993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:18.212950 5055 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:24:18.213516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:18.213014 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 08:24:19.212516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:19.212473 5055 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:24:19.212695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:19.212542 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 08:24:20.216680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:20.216652 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:20.217159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:20.217131 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:20.217924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:20.217899 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:24:24.924767 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:24:24.924719 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:24:45.240152 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.240103 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq"] Nov 02 08:24:45.240752 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.240731 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" Nov 02 08:24:45.240817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.240756 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" Nov 02 08:24:45.240900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.240889 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e83a04b-acfb-4583-b44c-a52d18e16588" containerName="istio-proxy" Nov 02 08:24:45.244174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.244153 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.340260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.340227 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e6ca80\"/\"distinct-2-w0-e6ca80-kserve-self-signed-certs\"" Nov 02 08:24:45.343529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.343497 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq"] Nov 02 08:24:45.349802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.349778 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.349905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.349846 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.349954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.349933 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.349991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.349965 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmp5n\" (UniqueName: \"kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.350122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.350087 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.350213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.350200 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.450915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.450880 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.450943 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451008 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451040 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451058 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmp5n\" (UniqueName: \"kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451124 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451471 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451526 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.451574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.451544 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.453300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.453281 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.453571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.453554 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.505212 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.505108 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmp5n\" (UniqueName: \"kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n\") pod \"distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.556759 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.556719 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:45.980625 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:45.980591 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq"] Nov 02 08:24:45.981527 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:24:45.981497 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30389c6_bcc7_4fd4_8498_11094b658732.slice/crio-1b876124ae9c0ef1667cc13faf8a8388d0dc0d3d63bde2c92c7ac7e88a809ba4 WatchSource:0}: Error finding container 1b876124ae9c0ef1667cc13faf8a8388d0dc0d3d63bde2c92c7ac7e88a809ba4: Status 404 returned error can't find the container with id 1b876124ae9c0ef1667cc13faf8a8388d0dc0d3d63bde2c92c7ac7e88a809ba4 Nov 02 08:24:46.500839 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:46.500796 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" event={"ID":"c30389c6-bcc7-4fd4-8498-11094b658732","Type":"ContainerStarted","Data":"5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912"} Nov 02 08:24:46.500839 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:46.500842 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" event={"ID":"c30389c6-bcc7-4fd4-8498-11094b658732","Type":"ContainerStarted","Data":"1b876124ae9c0ef1667cc13faf8a8388d0dc0d3d63bde2c92c7ac7e88a809ba4"} Nov 02 08:24:46.557699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:46.557662 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:46.567219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:46.567186 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:24:46.611197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:46.611111 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podStartSLOduration=1.611092312 podStartE2EDuration="1.611092312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:46.608676155 +0000 UTC m=+1133.699296574" watchObservedRunningTime="2026-11-02 08:24:46.611092312 +0000 UTC m=+1133.701712732" Nov 02 08:24:53.289433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:53.289398 5055 scope.go:122] "RemoveContainer" containerID="1726067eadc0b1837a2b34729dd41f3367eb0948e525d0862edccd415eb8fad1" Nov 02 08:24:53.300482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:53.300456 5055 scope.go:122] "RemoveContainer" containerID="00df14d9cbd04e9a520f13e7ec3ad2c9d343c9191a2c6ea43fab7690d0184ab1" Nov 02 08:24:53.309787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:53.309762 5055 scope.go:122] "RemoveContainer" containerID="00ef773284e609a59a5f8ab6b84d4f49b70ff2454a27ba2455fa81d83594f4e7" Nov 02 08:24:53.319517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:53.319496 5055 scope.go:122] "RemoveContainer" containerID="87c47daf229be605b04327821baadee69a6db47c35bfd74ab1736a5d6ae193b9" Nov 02 08:24:54.922233 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:24:54.922195 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:24:55.557207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:55.557175 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:55.570048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:55.570012 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:24:55.588365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:24:55.588336 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:25:05.106286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.106245 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb"] Nov 02 08:25:05.112364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.112337 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.211380 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.211343 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb"] Nov 02 08:25:05.230064 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230025 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230078 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230253 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230291 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cz7w\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230322 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230415 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230463 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230500 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.230777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.230638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.331526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331492 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.331736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331581 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.331736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331607 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.331736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331689 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.331736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331711 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cz7w\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331739 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331779 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331826 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.331883 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.332271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.332334 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.332413 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.332438 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.332806 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.332634 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.334645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.334622 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.334913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.334887 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.386974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.386882 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.387111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.387007 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cz7w\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.425894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.425860 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:05.862052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.862020 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb"] Nov 02 08:25:05.863408 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:25:05.863382 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53e4327_185b_4482_860c_784fc93cc695.slice/crio-e2c11556166e453e46be94480ab02844bd5e6aedb87a2bd7d7db22a113e3f3b9 WatchSource:0}: Error finding container e2c11556166e453e46be94480ab02844bd5e6aedb87a2bd7d7db22a113e3f3b9: Status 404 returned error can't find the container with id e2c11556166e453e46be94480ab02844bd5e6aedb87a2bd7d7db22a113e3f3b9 Nov 02 08:25:05.865853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.865815 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:25:05.865926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.865886 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:25:05.865926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:05.865912 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:25:06.664358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:06.664319 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" event={"ID":"a53e4327-185b-4482-860c-784fc93cc695","Type":"ContainerStarted","Data":"ecf82e517c9202998106d696502321f380e15fe900bf258dbb06bcfb9143bd27"} Nov 02 08:25:06.664777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:06.664407 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" event={"ID":"a53e4327-185b-4482-860c-784fc93cc695","Type":"ContainerStarted","Data":"e2c11556166e453e46be94480ab02844bd5e6aedb87a2bd7d7db22a113e3f3b9"} Nov 02 08:25:06.865236 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:06.865125 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podStartSLOduration=1.865107687 podStartE2EDuration="1.865107687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:25:06.818723535 +0000 UTC m=+1153.909343955" watchObservedRunningTime="2026-11-02 08:25:06.865107687 +0000 UTC m=+1153.955728108" Nov 02 08:25:07.426978 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:07.426932 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:07.428740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:07.428703 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 08:25:07.428882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:07.428776 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 08:25:08.427082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:08.427039 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 08:25:08.427566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:08.427115 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 08:25:09.427334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:09.427291 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 08:25:09.427902 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:09.427356 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 08:25:10.427089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:10.427052 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 08:25:10.427305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:10.427107 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 08:25:11.426336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:11.426297 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 08:25:11.426740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:11.426366 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 08:25:12.427343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:12.427292 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 08:25:12.427873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:12.427358 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 08:25:13.432086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:13.432055 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:13.432601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:13.432580 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:13.433363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:13.433341 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:25:17.597873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:17.597820 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:25:18.469298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.469252 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp"] Nov 02 08:25:18.474548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.474522 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538272 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538320 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538457 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538491 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538519 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fl5h\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538600 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538706 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538667 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538710 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.538744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.538733 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.562800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.562762 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp"] Nov 02 08:25:18.639523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639486 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.639523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639528 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639610 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639642 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639677 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fl5h\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639711 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639803 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639856 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639890 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.639916 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.640052 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.640371 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.640412 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.640662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.640598 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.642238 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.642218 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.642920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.642900 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.694780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.694749 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.694929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.694847 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fl5h\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:18.794655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:18.794568 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:19.229958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:19.229919 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp"] Nov 02 08:25:19.230806 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:25:19.230741 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce777614_2403_468d_bb9d_d7c43c5cab89.slice/crio-bb947a95f71b8720abe2c5ee917032d99300674cf7380b70fc31e476665baa4b WatchSource:0}: Error finding container bb947a95f71b8720abe2c5ee917032d99300674cf7380b70fc31e476665baa4b: Status 404 returned error can't find the container with id bb947a95f71b8720abe2c5ee917032d99300674cf7380b70fc31e476665baa4b Nov 02 08:25:19.233763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:19.233726 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:25:19.233868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:19.233808 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:25:19.233868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:19.233851 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:25:19.812126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:19.812085 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" event={"ID":"ce777614-2403-468d-bb9d-d7c43c5cab89","Type":"ContainerStarted","Data":"0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727"} Nov 02 08:25:19.812770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:19.812743 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" event={"ID":"ce777614-2403-468d-bb9d-d7c43c5cab89","Type":"ContainerStarted","Data":"bb947a95f71b8720abe2c5ee917032d99300674cf7380b70fc31e476665baa4b"} Nov 02 08:25:20.018005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:20.017948 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podStartSLOduration=2.017930728 podStartE2EDuration="2.017930728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:25:19.971182744 +0000 UTC m=+1167.061803163" watchObservedRunningTime="2026-11-02 08:25:20.017930728 +0000 UTC m=+1167.108551148" Nov 02 08:25:20.803167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:20.802314 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:20.805329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:20.805297 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:25:20.805499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:20.805359 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 08:25:21.796598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:21.796554 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:25:21.797261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:21.797210 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 08:25:22.795682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:22.795641 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:25:22.795994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:22.795960 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 08:25:23.801320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:23.801275 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:25:23.801947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:23.801912 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 08:25:24.807165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:24.805518 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:25:24.807165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:24.805603 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 08:25:24.934975 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:25:24.934935 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:25:25.795564 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:25.795521 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:25:25.795748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:25.795588 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 08:25:26.800050 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:26.800018 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:26.800476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:26.800335 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:26.801067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:26.801049 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:25:32.047711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:32.047672 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs"] Nov 02 08:25:32.094257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:32.094221 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4437474-lgdcs"] Nov 02 08:25:33.092199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:33.092168 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68736ea4-2030-44d9-be5e-b9fe8832c6d7" path="/var/lib/kubelet/pods/68736ea4-2030-44d9-be5e-b9fe8832c6d7/volumes" Nov 02 08:25:48.861622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:48.861578 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:25:53.256875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:53.256844 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:25:53.265070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:53.265042 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:25:54.920451 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:25:54.920407 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:25:55.044459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:55.044422 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4"] Nov 02 08:25:55.092593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:55.092561 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee09c02a-bmhb4"] Nov 02 08:25:57.042822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:57.042782 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4"] Nov 02 08:25:57.093488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:57.093452 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ca3f51-620a-47c2-ab1c-4146bc3e08fa" path="/var/lib/kubelet/pods/99ca3f51-620a-47c2-ab1c-4146bc3e08fa/volumes" Nov 02 08:25:57.093790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:57.093775 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a889d4a-9hhh4"] Nov 02 08:25:59.094392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:25:59.094354 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f10fe7c-5fb1-4b4e-a57e-061b48a676fb" path="/var/lib/kubelet/pods/8f10fe7c-5fb1-4b4e-a57e-061b48a676fb/volumes" Nov 02 08:26:13.169158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.169060 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d"] Nov 02 08:26:13.173094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.173069 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.269769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.269726 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:26:13.269769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.269759 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:26:13.270048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.269789 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:26:13.270048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.269936 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:26:13.273156 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.273118 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d"] Nov 02 08:26:13.284116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.284082 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.284294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.284266 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.284342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.284331 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdtz\" (UniqueName: \"kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.385579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.385544 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdtz\" (UniqueName: \"kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.385762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.385645 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.385762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.385739 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.386365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.386342 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.388105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.388085 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.439006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.438929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdtz\" (UniqueName: \"kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz\") pod \"maas-api-revoke-sub-82099558-jt57d\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.486866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.486814 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:13.855356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:13.855326 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d"] Nov 02 08:26:13.857754 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:26:13.857717 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3059ceff_af81_46fe_b8af_c7bb7f96293c.slice/crio-5a1b4c2dd4ad88a510137a1d746c76c5477198184dea71235ee11cb628910cc8 WatchSource:0}: Error finding container 5a1b4c2dd4ad88a510137a1d746c76c5477198184dea71235ee11cb628910cc8: Status 404 returned error can't find the container with id 5a1b4c2dd4ad88a510137a1d746c76c5477198184dea71235ee11cb628910cc8 Nov 02 08:26:14.213961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:14.213904 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" event={"ID":"3059ceff-af81-46fe-b8af-c7bb7f96293c","Type":"ContainerStarted","Data":"537330bad4206f5338f7e578ef317c5228dfebff33134b06a6b4d17a97ec5787"} Nov 02 08:26:14.214570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:14.213975 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" event={"ID":"3059ceff-af81-46fe-b8af-c7bb7f96293c","Type":"ContainerStarted","Data":"5a1b4c2dd4ad88a510137a1d746c76c5477198184dea71235ee11cb628910cc8"} Nov 02 08:26:14.320982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:14.320925 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" podStartSLOduration=1.320907609 podStartE2EDuration="1.320907609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:26:14.318463559 +0000 UTC m=+1221.409083977" watchObservedRunningTime="2026-11-02 08:26:14.320907609 +0000 UTC m=+1221.411528029" Nov 02 08:26:15.043515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:15.043431 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4"] Nov 02 08:26:15.093865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:15.093832 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57c420b-g28n4"] Nov 02 08:26:15.224958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:15.224922 5055 generic.go:356] "Generic (PLEG): container finished" podID="3059ceff-af81-46fe-b8af-c7bb7f96293c" containerID="537330bad4206f5338f7e578ef317c5228dfebff33134b06a6b4d17a97ec5787" exitCode=0 Nov 02 08:26:15.225373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:15.225001 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" event={"ID":"3059ceff-af81-46fe-b8af-c7bb7f96293c","Type":"ContainerDied","Data":"537330bad4206f5338f7e578ef317c5228dfebff33134b06a6b4d17a97ec5787"} Nov 02 08:26:16.402638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.402612 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:16.527072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.526983 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz\" (UniqueName: \"kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz\") pod \"3059ceff-af81-46fe-b8af-c7bb7f96293c\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " Nov 02 08:26:16.527259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.527134 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca\") pod \"3059ceff-af81-46fe-b8af-c7bb7f96293c\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " Nov 02 08:26:16.527305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.527265 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca\") pod \"3059ceff-af81-46fe-b8af-c7bb7f96293c\" (UID: \"3059ceff-af81-46fe-b8af-c7bb7f96293c\") " Nov 02 08:26:16.527502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.527473 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca" pod "3059ceff-af81-46fe-b8af-c7bb7f96293c" (UID: "3059ceff-af81-46fe-b8af-c7bb7f96293c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:26:16.527859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.527839 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:16.529925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.529889 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz" pod "3059ceff-af81-46fe-b8af-c7bb7f96293c" (UID: "3059ceff-af81-46fe-b8af-c7bb7f96293c"). InnerVolumeSpecName "kube-api-access-mxdtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:16.529925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.529913 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca" pod "3059ceff-af81-46fe-b8af-c7bb7f96293c" (UID: "3059ceff-af81-46fe-b8af-c7bb7f96293c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:16.629430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.629390 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3059ceff-af81-46fe-b8af-c7bb7f96293c-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:16.629430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:16.629426 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdtz\" (UniqueName: \"kubernetes.io/projected/3059ceff-af81-46fe-b8af-c7bb7f96293c-kube-api-access-mxdtz\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:17.092252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:17.092217 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae8589d-07ad-4186-8353-90b9a8dd25f4" path="/var/lib/kubelet/pods/7ae8589d-07ad-4186-8353-90b9a8dd25f4/volumes" Nov 02 08:26:17.240889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:17.240862 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" Nov 02 08:26:17.240889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:17.240875 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d" event={"ID":"3059ceff-af81-46fe-b8af-c7bb7f96293c","Type":"ContainerDied","Data":"5a1b4c2dd4ad88a510137a1d746c76c5477198184dea71235ee11cb628910cc8"} Nov 02 08:26:17.241116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:17.240919 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a1b4c2dd4ad88a510137a1d746c76c5477198184dea71235ee11cb628910cc8" Nov 02 08:26:23.469615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:23.469576 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:26:24.053613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:24.053568 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7"] Nov 02 08:26:24.098106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:24.098066 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50bc390-ms5f7"] Nov 02 08:26:24.952479 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:26:24.952442 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:26:25.098184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:25.098123 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184d9e8a-6044-454a-b583-9ab4cf644045" path="/var/lib/kubelet/pods/184d9e8a-6044-454a-b583-9ab4cf644045/volumes" Nov 02 08:26:27.044002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:27.043966 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj"] Nov 02 08:26:27.092494 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:27.092462 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620f49b9-tmxbj"] Nov 02 08:26:29.098978 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:29.098888 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270e2570-d15e-4ae7-9eed-d64e90ee5407" path="/var/lib/kubelet/pods/270e2570-d15e-4ae7-9eed-d64e90ee5407/volumes" Nov 02 08:26:31.496506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.496464 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn"] Nov 02 08:26:31.497304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.497281 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3059ceff-af81-46fe-b8af-c7bb7f96293c" containerName="revoke-keys" Nov 02 08:26:31.497304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.497304 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3059ceff-af81-46fe-b8af-c7bb7f96293c" containerName="revoke-keys" Nov 02 08:26:31.497501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.497487 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3059ceff-af81-46fe-b8af-c7bb7f96293c" containerName="revoke-keys" Nov 02 08:26:31.502309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.502287 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.552226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.552185 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:26:31.553801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.552462 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:26:31.555472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.555437 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn"] Nov 02 08:26:31.567908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.567864 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.568321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.568281 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.568449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.568336 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgwg\" (UniqueName: \"kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.596922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.596885 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:26:31.597191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.596907 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:26:31.669521 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.669480 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.669725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.669577 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.669725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.669597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgwg\" (UniqueName: \"kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.670311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.670288 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.672020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.671998 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.724931 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.724894 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgwg\" (UniqueName: \"kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg\") pod \"maas-api-revoke-sub-75a559b8-dhdsn\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:31.821755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:31.821647 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:32.196090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:32.196056 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn"] Nov 02 08:26:32.196340 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:26:32.196314 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6f91e0_2360_420b_9d63_ae2c801ded28.slice/crio-f2fb75c81d9252028332e272cb827722f44634d183c91ad439846fd557fcda9e WatchSource:0}: Error finding container f2fb75c81d9252028332e272cb827722f44634d183c91ad439846fd557fcda9e: Status 404 returned error can't find the container with id f2fb75c81d9252028332e272cb827722f44634d183c91ad439846fd557fcda9e Nov 02 08:26:32.354489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:32.354449 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" event={"ID":"2f6f91e0-2360-420b-9d63-ae2c801ded28","Type":"ContainerStarted","Data":"f2fb75c81d9252028332e272cb827722f44634d183c91ad439846fd557fcda9e"} Nov 02 08:26:33.365299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:33.365199 5055 generic.go:356] "Generic (PLEG): container finished" podID="2f6f91e0-2360-420b-9d63-ae2c801ded28" containerID="8529463f474fd5f863420fc57074fd3e3cb3f22e6f470d0379517939403e4e29" exitCode=0 Nov 02 08:26:33.365299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:33.365259 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" event={"ID":"2f6f91e0-2360-420b-9d63-ae2c801ded28","Type":"ContainerDied","Data":"8529463f474fd5f863420fc57074fd3e3cb3f22e6f470d0379517939403e4e29"} Nov 02 08:26:34.535582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.535552 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:34.609194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.609154 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca\") pod \"2f6f91e0-2360-420b-9d63-ae2c801ded28\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " Nov 02 08:26:34.609395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.609205 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg\" (UniqueName: \"kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg\") pod \"2f6f91e0-2360-420b-9d63-ae2c801ded28\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " Nov 02 08:26:34.609395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.609373 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca\") pod \"2f6f91e0-2360-420b-9d63-ae2c801ded28\" (UID: \"2f6f91e0-2360-420b-9d63-ae2c801ded28\") " Nov 02 08:26:34.609508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.609434 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca" pod "2f6f91e0-2360-420b-9d63-ae2c801ded28" (UID: "2f6f91e0-2360-420b-9d63-ae2c801ded28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:26:34.609875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.609853 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:34.611264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.611243 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca" pod "2f6f91e0-2360-420b-9d63-ae2c801ded28" (UID: "2f6f91e0-2360-420b-9d63-ae2c801ded28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:34.611406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.611391 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg" pod "2f6f91e0-2360-420b-9d63-ae2c801ded28" (UID: "2f6f91e0-2360-420b-9d63-ae2c801ded28"). InnerVolumeSpecName "kube-api-access-cbgwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:34.711504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.711471 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgwg\" (UniqueName: \"kubernetes.io/projected/2f6f91e0-2360-420b-9d63-ae2c801ded28-kube-api-access-cbgwg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:34.711504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:34.711500 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6f91e0-2360-420b-9d63-ae2c801ded28-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:35.386446 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:35.386357 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" Nov 02 08:26:35.386446 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:35.386388 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn" event={"ID":"2f6f91e0-2360-420b-9d63-ae2c801ded28","Type":"ContainerDied","Data":"f2fb75c81d9252028332e272cb827722f44634d183c91ad439846fd557fcda9e"} Nov 02 08:26:35.386446 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:35.386442 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2fb75c81d9252028332e272cb827722f44634d183c91ad439846fd557fcda9e" Nov 02 08:26:39.573559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.573520 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r"] Nov 02 08:26:39.574126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.574110 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2f6f91e0-2360-420b-9d63-ae2c801ded28" containerName="revoke-keys" Nov 02 08:26:39.574192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.574126 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6f91e0-2360-420b-9d63-ae2c801ded28" containerName="revoke-keys" Nov 02 08:26:39.574275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.574264 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6f91e0-2360-420b-9d63-ae2c801ded28" containerName="revoke-keys" Nov 02 08:26:39.577979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.577956 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.632860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.632817 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r"] Nov 02 08:26:39.673180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.673131 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:26:39.673376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.673131 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:26:39.673376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.673225 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:26:39.673376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.673360 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:26:39.676686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.676654 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf67l\" (UniqueName: \"kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.676992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.676963 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.677069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.677013 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.778638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.778597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.778638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.778639 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.778882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.778696 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf67l\" (UniqueName: \"kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.779519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.779493 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.781558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.781523 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.831633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.831558 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf67l\" (UniqueName: \"kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l\") pod \"maas-api-revoke-sub-e07aef76-x869r\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:39.896549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:39.896510 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:40.276359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:40.276311 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r"] Nov 02 08:26:40.280067 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:26:40.280032 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84073ccf_5668_46ed_ba88_67c80ed180ad.slice/crio-abe6b2ed5d742738a492009f4dbdb0f08444169a9b47c1875c753c002ef3709b WatchSource:0}: Error finding container abe6b2ed5d742738a492009f4dbdb0f08444169a9b47c1875c753c002ef3709b: Status 404 returned error can't find the container with id abe6b2ed5d742738a492009f4dbdb0f08444169a9b47c1875c753c002ef3709b Nov 02 08:26:40.424500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:40.424459 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" event={"ID":"84073ccf-5668-46ed-ba88-67c80ed180ad","Type":"ContainerStarted","Data":"abe6b2ed5d742738a492009f4dbdb0f08444169a9b47c1875c753c002ef3709b"} Nov 02 08:26:41.434497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:41.434406 5055 generic.go:356] "Generic (PLEG): container finished" podID="84073ccf-5668-46ed-ba88-67c80ed180ad" containerID="6e833ef5a7cd5bd86df6c91f6acf4babee983dea8a694e49c8f794f70f814020" exitCode=0 Nov 02 08:26:41.434497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:41.434462 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" event={"ID":"84073ccf-5668-46ed-ba88-67c80ed180ad","Type":"ContainerDied","Data":"6e833ef5a7cd5bd86df6c91f6acf4babee983dea8a694e49c8f794f70f814020"} Nov 02 08:26:42.615857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.615824 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:42.722994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.722961 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca\") pod \"84073ccf-5668-46ed-ba88-67c80ed180ad\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " Nov 02 08:26:42.723264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.723008 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca\") pod \"84073ccf-5668-46ed-ba88-67c80ed180ad\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " Nov 02 08:26:42.723264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.723134 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l\" (UniqueName: \"kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l\") pod \"84073ccf-5668-46ed-ba88-67c80ed180ad\" (UID: \"84073ccf-5668-46ed-ba88-67c80ed180ad\") " Nov 02 08:26:42.723716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.723675 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca" pod "84073ccf-5668-46ed-ba88-67c80ed180ad" (UID: "84073ccf-5668-46ed-ba88-67c80ed180ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:26:42.726119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.726068 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca" pod "84073ccf-5668-46ed-ba88-67c80ed180ad" (UID: "84073ccf-5668-46ed-ba88-67c80ed180ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:42.726296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.726183 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l" pod "84073ccf-5668-46ed-ba88-67c80ed180ad" (UID: "84073ccf-5668-46ed-ba88-67c80ed180ad"). InnerVolumeSpecName "kube-api-access-jf67l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:42.825527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.825476 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:42.825527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.825520 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84073ccf-5668-46ed-ba88-67c80ed180ad-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:42.825527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:42.825532 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf67l\" (UniqueName: \"kubernetes.io/projected/84073ccf-5668-46ed-ba88-67c80ed180ad-kube-api-access-jf67l\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:43.450687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:43.450657 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:26:43.450863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:43.450687 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" event={"ID":"84073ccf-5668-46ed-ba88-67c80ed180ad","Type":"ContainerDied","Data":"abe6b2ed5d742738a492009f4dbdb0f08444169a9b47c1875c753c002ef3709b"} Nov 02 08:26:43.450863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:43.450744 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abe6b2ed5d742738a492009f4dbdb0f08444169a9b47c1875c753c002ef3709b" Nov 02 08:26:47.574266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.574213 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb"] Nov 02 08:26:47.575364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.575134 5055 cpu_manager.go:414] "containerMap: removing container" podUID="84073ccf-5668-46ed-ba88-67c80ed180ad" containerName="revoke-keys" Nov 02 08:26:47.575364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.575188 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84073ccf-5668-46ed-ba88-67c80ed180ad" containerName="revoke-keys" Nov 02 08:26:47.575546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.575377 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="84073ccf-5668-46ed-ba88-67c80ed180ad" containerName="revoke-keys" Nov 02 08:26:47.579436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.579400 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.628584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.628484 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:26:47.675693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.674716 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:26:47.675693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.674716 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:26:47.675693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.674852 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:26:47.678269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.678238 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb"] Nov 02 08:26:47.692780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.692735 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.692976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.692954 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrz4b\" (UniqueName: \"kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.693079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.693062 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.794774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.794730 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrz4b\" (UniqueName: \"kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.794993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.794801 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.794993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.794928 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.795602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.795578 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.797318 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.797294 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.847780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.847699 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrz4b\" (UniqueName: \"kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b\") pod \"maas-api-revoke-sub-66bd1e85-thtfb\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:47.898971 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:47.898924 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:48.294800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:48.294766 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb"] Nov 02 08:26:48.487301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:48.487266 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" event={"ID":"740fe1fe-8ef5-47cf-a742-3ee60f946e18","Type":"ContainerStarted","Data":"1c3126f44cd28111c941fa059716c664b2af6502c5569d6b5cc318c1ff6c1bd0"} Nov 02 08:26:49.498194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:49.498122 5055 generic.go:356] "Generic (PLEG): container finished" podID="740fe1fe-8ef5-47cf-a742-3ee60f946e18" containerID="663b8af2bd0671be76761ba802d80353a89adb74eec388702610ba412b37d208" exitCode=0 Nov 02 08:26:49.498604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:49.498216 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" event={"ID":"740fe1fe-8ef5-47cf-a742-3ee60f946e18","Type":"ContainerDied","Data":"663b8af2bd0671be76761ba802d80353a89adb74eec388702610ba412b37d208"} Nov 02 08:26:50.313671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.313631 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl"] Nov 02 08:26:50.318576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.318550 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.369516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.369466 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl"] Nov 02 08:26:50.433732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.433700 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.433942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.433766 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.434067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.434041 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzssv\" (UniqueName: \"kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.535601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.535562 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzssv\" (UniqueName: \"kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.536069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.535647 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.536069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.535730 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.536474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.536447 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.538426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.538400 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.603767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.603690 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzssv\" (UniqueName: \"kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv\") pod \"maas-api-revoke-sub-22e4e003-f2mhl\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.636747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.636703 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:50.681387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.681354 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:50.738527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.738493 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca\") pod \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " Nov 02 08:26:50.738718 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.738555 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b\" (UniqueName: \"kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b\") pod \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " Nov 02 08:26:50.738718 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.738607 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca\") pod \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\" (UID: \"740fe1fe-8ef5-47cf-a742-3ee60f946e18\") " Nov 02 08:26:50.739365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.739333 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca" pod "740fe1fe-8ef5-47cf-a742-3ee60f946e18" (UID: "740fe1fe-8ef5-47cf-a742-3ee60f946e18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:26:50.741534 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.741511 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca" pod "740fe1fe-8ef5-47cf-a742-3ee60f946e18" (UID: "740fe1fe-8ef5-47cf-a742-3ee60f946e18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:50.741607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.741552 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b" pod "740fe1fe-8ef5-47cf-a742-3ee60f946e18" (UID: "740fe1fe-8ef5-47cf-a742-3ee60f946e18"). InnerVolumeSpecName "kube-api-access-wrz4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:50.839982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.839899 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:50.839982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.839930 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrz4b\" (UniqueName: \"kubernetes.io/projected/740fe1fe-8ef5-47cf-a742-3ee60f946e18-kube-api-access-wrz4b\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:50.839982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:50.839941 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740fe1fe-8ef5-47cf-a742-3ee60f946e18-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.012173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.012132 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl"] Nov 02 08:26:51.014712 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:26:51.014678 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0763829_af25_4494_98d8_72acf1aeeac9.slice/crio-d851e19f1e1ad10059736bde98f1cfe24d3df09021d8c09bd26d7e91e50ffda6 WatchSource:0}: Error finding container d851e19f1e1ad10059736bde98f1cfe24d3df09021d8c09bd26d7e91e50ffda6: Status 404 returned error can't find the container with id d851e19f1e1ad10059736bde98f1cfe24d3df09021d8c09bd26d7e91e50ffda6 Nov 02 08:26:51.518967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.518919 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" event={"ID":"740fe1fe-8ef5-47cf-a742-3ee60f946e18","Type":"ContainerDied","Data":"1c3126f44cd28111c941fa059716c664b2af6502c5569d6b5cc318c1ff6c1bd0"} Nov 02 08:26:51.518967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.518958 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb" Nov 02 08:26:51.519270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.519007 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c3126f44cd28111c941fa059716c664b2af6502c5569d6b5cc318c1ff6c1bd0" Nov 02 08:26:51.521222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.521188 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" event={"ID":"f0763829-af25-4494-98d8-72acf1aeeac9","Type":"ContainerStarted","Data":"2798c109520437bb8719961e34b1900ff2617dcd6e9f8d13166e59673c1c02fc"} Nov 02 08:26:51.521615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.521250 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" event={"ID":"f0763829-af25-4494-98d8-72acf1aeeac9","Type":"ContainerStarted","Data":"d851e19f1e1ad10059736bde98f1cfe24d3df09021d8c09bd26d7e91e50ffda6"} Nov 02 08:26:51.627497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:51.627443 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" podStartSLOduration=1.627429317 podStartE2EDuration="1.627429317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:26:51.626028934 +0000 UTC m=+1258.716649352" watchObservedRunningTime="2026-11-02 08:26:51.627429317 +0000 UTC m=+1258.718049735" Nov 02 08:26:52.536655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:52.536570 5055 generic.go:356] "Generic (PLEG): container finished" podID="f0763829-af25-4494-98d8-72acf1aeeac9" containerID="2798c109520437bb8719961e34b1900ff2617dcd6e9f8d13166e59673c1c02fc" exitCode=0 Nov 02 08:26:52.536903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:52.536779 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" event={"ID":"f0763829-af25-4494-98d8-72acf1aeeac9","Type":"ContainerDied","Data":"2798c109520437bb8719961e34b1900ff2617dcd6e9f8d13166e59673c1c02fc"} Nov 02 08:26:53.375710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.375678 5055 scope.go:122] "RemoveContainer" containerID="912eb4cbd408be863be0cd8a2b6d217f097da0eba2ba677829fcacedbf3110c9" Nov 02 08:26:53.391782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.391747 5055 scope.go:122] "RemoveContainer" containerID="69e9225cf40f664b1ffd60bc3d88078cc20de87ca4e9a3b3d78bec7a4795b601" Nov 02 08:26:53.713174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.713130 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:53.881164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.876622 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca\") pod \"f0763829-af25-4494-98d8-72acf1aeeac9\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " Nov 02 08:26:53.881164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.876704 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv\" (UniqueName: \"kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv\") pod \"f0763829-af25-4494-98d8-72acf1aeeac9\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " Nov 02 08:26:53.881164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.876852 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca\") pod \"f0763829-af25-4494-98d8-72acf1aeeac9\" (UID: \"f0763829-af25-4494-98d8-72acf1aeeac9\") " Nov 02 08:26:53.881164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.878385 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca" pod "f0763829-af25-4494-98d8-72acf1aeeac9" (UID: "f0763829-af25-4494-98d8-72acf1aeeac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:26:53.911177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.910031 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca" pod "f0763829-af25-4494-98d8-72acf1aeeac9" (UID: "f0763829-af25-4494-98d8-72acf1aeeac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:53.943176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.940949 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv" pod "f0763829-af25-4494-98d8-72acf1aeeac9" (UID: "f0763829-af25-4494-98d8-72acf1aeeac9"). InnerVolumeSpecName "kube-api-access-hzssv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:53.982809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.981716 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzssv\" (UniqueName: \"kubernetes.io/projected/f0763829-af25-4494-98d8-72acf1aeeac9-kube-api-access-hzssv\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:53.982809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.981772 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:53.982809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:53.981788 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0763829-af25-4494-98d8-72acf1aeeac9-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:54.602680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:54.601999 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" event={"ID":"f0763829-af25-4494-98d8-72acf1aeeac9","Type":"ContainerDied","Data":"d851e19f1e1ad10059736bde98f1cfe24d3df09021d8c09bd26d7e91e50ffda6"} Nov 02 08:26:54.602680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:54.602077 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d851e19f1e1ad10059736bde98f1cfe24d3df09021d8c09bd26d7e91e50ffda6" Nov 02 08:26:54.602680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:26:54.602655 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl" Nov 02 08:26:54.926582 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:26:54.926480 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:27:06.459382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.459343 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f"] Nov 02 08:27:06.459930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.459915 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f0763829-af25-4494-98d8-72acf1aeeac9" containerName="revoke-keys" Nov 02 08:27:06.459930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.459931 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0763829-af25-4494-98d8-72acf1aeeac9" containerName="revoke-keys" Nov 02 08:27:06.460014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.459945 5055 cpu_manager.go:414] "containerMap: removing container" podUID="740fe1fe-8ef5-47cf-a742-3ee60f946e18" containerName="revoke-keys" Nov 02 08:27:06.460014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.459950 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740fe1fe-8ef5-47cf-a742-3ee60f946e18" containerName="revoke-keys" Nov 02 08:27:06.460076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.460039 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="740fe1fe-8ef5-47cf-a742-3ee60f946e18" containerName="revoke-keys" Nov 02 08:27:06.460076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.460059 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0763829-af25-4494-98d8-72acf1aeeac9" containerName="revoke-keys" Nov 02 08:27:06.463896 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.463877 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.513053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.513006 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:27:06.545462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.545428 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:27:06.545624 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.545428 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:27:06.552174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.552109 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.552343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.552205 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldgw\" (UniqueName: \"kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.552343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.552265 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.558851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.558822 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:27:06.563089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.563062 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f"] Nov 02 08:27:06.653572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.653535 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.653750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.653601 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldgw\" (UniqueName: \"kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.653750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.653653 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.654266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.654235 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.656009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.655990 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.706339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.706304 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldgw\" (UniqueName: \"kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw\") pod \"maas-api-revoke-sub-3aa7e00b-kmb7f\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:06.777093 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:06.777005 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:07.151232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:07.151199 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f"] Nov 02 08:27:07.152059 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:07.152026 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a26b0bd_c8e0_4b4d_8f21_33dad744eb2e.slice/crio-7ba1b08d83302333a8d5c4631c943839fa0cf2e0933c473b2507183f3af7099f WatchSource:0}: Error finding container 7ba1b08d83302333a8d5c4631c943839fa0cf2e0933c473b2507183f3af7099f: Status 404 returned error can't find the container with id 7ba1b08d83302333a8d5c4631c943839fa0cf2e0933c473b2507183f3af7099f Nov 02 08:27:07.696210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:07.696170 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" event={"ID":"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e","Type":"ContainerStarted","Data":"31613fa51c949668e9866e6771ed2d2f3be8f5322c714275628ec743a250b17d"} Nov 02 08:27:07.696707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:07.696230 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" event={"ID":"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e","Type":"ContainerStarted","Data":"7ba1b08d83302333a8d5c4631c943839fa0cf2e0933c473b2507183f3af7099f"} Nov 02 08:27:07.802974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:07.802859 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" podStartSLOduration=1.8028431999999999 podStartE2EDuration="1.8028432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:07.801864571 +0000 UTC m=+1274.892484990" watchObservedRunningTime="2026-11-02 08:27:07.8028432 +0000 UTC m=+1274.893463619" Nov 02 08:27:08.705700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:08.705665 5055 generic.go:356] "Generic (PLEG): container finished" podID="0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" containerID="31613fa51c949668e9866e6771ed2d2f3be8f5322c714275628ec743a250b17d" exitCode=0 Nov 02 08:27:08.706254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:08.705781 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" event={"ID":"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e","Type":"ContainerDied","Data":"31613fa51c949668e9866e6771ed2d2f3be8f5322c714275628ec743a250b17d"} Nov 02 08:27:09.884155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.884119 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:09.993864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.993781 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca\") pod \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " Nov 02 08:27:09.994221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.994186 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca" pod "0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" (UID: "0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:09.994329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.994198 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca\") pod \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " Nov 02 08:27:09.994426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.994409 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw\" (UniqueName: \"kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw\") pod \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\" (UID: \"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e\") " Nov 02 08:27:09.994898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.994874 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:09.997314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.997281 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca" pod "0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" (UID: "0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:09.997412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:09.997387 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw" pod "0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" (UID: "0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e"). InnerVolumeSpecName "kube-api-access-wldgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:10.096045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:10.096006 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:10.096045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:10.096038 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldgw\" (UniqueName: \"kubernetes.io/projected/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e-kube-api-access-wldgw\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:10.721126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:10.721095 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" Nov 02 08:27:10.721304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:10.721131 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f" event={"ID":"0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e","Type":"ContainerDied","Data":"7ba1b08d83302333a8d5c4631c943839fa0cf2e0933c473b2507183f3af7099f"} Nov 02 08:27:10.721304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:10.721202 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba1b08d83302333a8d5c4631c943839fa0cf2e0933c473b2507183f3af7099f" Nov 02 08:27:13.537064 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.536990 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg"] Nov 02 08:27:13.537835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.537805 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" containerName="revoke-keys" Nov 02 08:27:13.537835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.537831 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" containerName="revoke-keys" Nov 02 08:27:13.538022 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.537958 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" containerName="revoke-keys" Nov 02 08:27:13.541765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.541732 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.630799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.630763 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg"] Nov 02 08:27:13.644729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644693 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.644878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644752 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.644878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644784 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.644878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644812 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.644878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644834 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.645045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644880 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.645045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644904 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcs85\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.645045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644939 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.645045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.644974 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745623 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745669 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745743 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745779 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745805 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745825 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745846 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745898 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.745994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.745924 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcs85\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.746556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.746529 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.746742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.746726 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.747250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.747226 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.747366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.747335 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.747600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.747578 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.749048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.749029 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.749677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.749652 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.799109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.799031 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.799292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.799211 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcs85\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:13.857398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:13.857361 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:14.285786 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.285730 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg"] Nov 02 08:27:14.286947 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:14.286906 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3674ca_ed1d_4b6a_aba7_8d5202a4d2dd.slice/crio-f5275781982ec321509705e49635898c6ac3c73a404ac5df5546f2d0875e6c7e WatchSource:0}: Error finding container f5275781982ec321509705e49635898c6ac3c73a404ac5df5546f2d0875e6c7e: Status 404 returned error can't find the container with id f5275781982ec321509705e49635898c6ac3c73a404ac5df5546f2d0875e6c7e Nov 02 08:27:14.289794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.289755 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:27:14.289875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.289841 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:27:14.289919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.289884 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:27:14.754388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.754335 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" event={"ID":"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd","Type":"ContainerStarted","Data":"ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181"} Nov 02 08:27:14.754753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.754420 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" event={"ID":"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd","Type":"ContainerStarted","Data":"f5275781982ec321509705e49635898c6ac3c73a404ac5df5546f2d0875e6c7e"} Nov 02 08:27:14.857496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.857460 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:14.864389 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:14.864344 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podStartSLOduration=1.86433059 podStartE2EDuration="1.86433059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:14.861867982 +0000 UTC m=+1281.952488402" watchObservedRunningTime="2026-11-02 08:27:14.86433059 +0000 UTC m=+1281.954951011" Nov 02 08:27:15.860067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:15.860027 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:15.860530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:15.860101 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:16.858048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:16.858002 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:16.858349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:16.858065 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:17.212297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.212259 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww"] Nov 02 08:27:17.220987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.220957 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.266956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.266922 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww"] Nov 02 08:27:17.308495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.308464 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:27:17.308778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.308763 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:27:17.308990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.308976 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:27:17.318317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.318287 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:27:17.319252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.319223 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqg8\" (UniqueName: \"kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.319496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.319478 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.319702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.319686 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.420910 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.420869 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqg8\" (UniqueName: \"kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.421121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.420978 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.421121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.421096 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.421742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.421715 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.424107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.424082 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.474991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.474911 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqg8\" (UniqueName: \"kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8\") pod \"maas-api-revoke-sub-242e3caa-7txww\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.541132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.541085 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:17.858154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.858052 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:17.858154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.858112 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:17.956864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:17.956833 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww"] Nov 02 08:27:17.957993 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:17.957963 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ed46ba_8dea_4868_a335_147ac8d8a683.slice/crio-d268d8684ab93ef1ddcc6238e4fbbe785301696b96c23e04a376ee4893915442 WatchSource:0}: Error finding container d268d8684ab93ef1ddcc6238e4fbbe785301696b96c23e04a376ee4893915442: Status 404 returned error can't find the container with id d268d8684ab93ef1ddcc6238e4fbbe785301696b96c23e04a376ee4893915442 Nov 02 08:27:18.786768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:18.786722 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" event={"ID":"39ed46ba-8dea-4868-a335-147ac8d8a683","Type":"ContainerStarted","Data":"da272c0ce48ea2eb489b040a4d0e1b8cc3a1e35662cbaa02c34c716c4d063672"} Nov 02 08:27:18.787408 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:18.787362 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" event={"ID":"39ed46ba-8dea-4868-a335-147ac8d8a683","Type":"ContainerStarted","Data":"d268d8684ab93ef1ddcc6238e4fbbe785301696b96c23e04a376ee4893915442"} Nov 02 08:27:18.858509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:18.858474 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:18.858694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:18.858529 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:18.906168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:18.903518 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" podStartSLOduration=1.9034998010000002 podStartE2EDuration="1.903499801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:18.901019027 +0000 UTC m=+1285.991639441" watchObservedRunningTime="2026-11-02 08:27:18.903499801 +0000 UTC m=+1285.994120221" Nov 02 08:27:19.816415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:19.816379 5055 generic.go:356] "Generic (PLEG): container finished" podID="39ed46ba-8dea-4868-a335-147ac8d8a683" containerID="da272c0ce48ea2eb489b040a4d0e1b8cc3a1e35662cbaa02c34c716c4d063672" exitCode=0 Nov 02 08:27:19.816909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:19.816473 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" event={"ID":"39ed46ba-8dea-4868-a335-147ac8d8a683","Type":"ContainerDied","Data":"da272c0ce48ea2eb489b040a4d0e1b8cc3a1e35662cbaa02c34c716c4d063672"} Nov 02 08:27:19.857857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:19.857818 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:19.858048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:19.857873 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:20.526595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.526548 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq"] Nov 02 08:27:20.531072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.531043 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.661172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.642665 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cba6f9af-openshift-default-dockercfg-lc888\"" Nov 02 08:27:20.681321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.665569 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:27:20.684996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.684953 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.685323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.685297 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.685453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.685438 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.685588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.685572 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.685695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.685680 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.685857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.685841 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.685977 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.685960 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26g7j\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.686169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.686134 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.686287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.686272 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.754388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.753744 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq"] Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792665 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792717 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792746 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792795 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792824 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792900 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792932 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26g7j\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.792993 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.793021 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.793847 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.794689 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.795420 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.795894 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.799241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.796201 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.826794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.826719 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.827831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.827769 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.847227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.847200 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.850393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.850109 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26g7j\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:20.860173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.860101 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:20.860325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:20.860205 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:21.015785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.015744 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:21.105219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.103703 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca\") pod \"39ed46ba-8dea-4868-a335-147ac8d8a683\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " Nov 02 08:27:21.105219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.103757 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca\") pod \"39ed46ba-8dea-4868-a335-147ac8d8a683\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " Nov 02 08:27:21.105219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.103854 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8\" (UniqueName: \"kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8\") pod \"39ed46ba-8dea-4868-a335-147ac8d8a683\" (UID: \"39ed46ba-8dea-4868-a335-147ac8d8a683\") " Nov 02 08:27:21.110192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.110098 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca" pod "39ed46ba-8dea-4868-a335-147ac8d8a683" (UID: "39ed46ba-8dea-4868-a335-147ac8d8a683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:21.147165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.146308 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:21.194416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.194317 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8" pod "39ed46ba-8dea-4868-a335-147ac8d8a683" (UID: "39ed46ba-8dea-4868-a335-147ac8d8a683"). InnerVolumeSpecName "kube-api-access-4fqg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:21.196348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.196291 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca" pod "39ed46ba-8dea-4868-a335-147ac8d8a683" (UID: "39ed46ba-8dea-4868-a335-147ac8d8a683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:21.206929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.206215 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:21.206929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.206250 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ed46ba-8dea-4868-a335-147ac8d8a683-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:21.206929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.206266 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fqg8\" (UniqueName: \"kubernetes.io/projected/39ed46ba-8dea-4868-a335-147ac8d8a683-kube-api-access-4fqg8\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:21.629482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.629410 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq"] Nov 02 08:27:21.631953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.631904 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:27:21.632071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.632001 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:27:21.632071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.632045 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:27:21.856160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.855722 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" Nov 02 08:27:21.856160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.855812 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww" event={"ID":"39ed46ba-8dea-4868-a335-147ac8d8a683","Type":"ContainerDied","Data":"d268d8684ab93ef1ddcc6238e4fbbe785301696b96c23e04a376ee4893915442"} Nov 02 08:27:21.856160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.855853 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d268d8684ab93ef1ddcc6238e4fbbe785301696b96c23e04a376ee4893915442" Nov 02 08:27:21.858491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.857764 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 08:27:21.858491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.857820 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 08:27:21.858688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.858544 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" event={"ID":"0dede8b3-0d74-4932-9aa0-248f41ab23ea","Type":"ContainerStarted","Data":"eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3"} Nov 02 08:27:21.858688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.858594 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" event={"ID":"0dede8b3-0d74-4932-9aa0-248f41ab23ea","Type":"ContainerStarted","Data":"4c57ad5e42908e91f6965666001ccaa3970cb79ab64c6458544a6f0d2f441f8e"} Nov 02 08:27:21.968607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:21.968542 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podStartSLOduration=1.968522835 podStartE2EDuration="1.968522835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:21.966944211 +0000 UTC m=+1289.057564666" watchObservedRunningTime="2026-11-02 08:27:21.968522835 +0000 UTC m=+1289.059143254" Nov 02 08:27:22.151810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:22.150611 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:22.151810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:22.150742 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:27:22.151810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:22.150787 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:27:22.862439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:22.862408 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:22.862958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:22.862939 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:22.863851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:22.863827 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:27:23.152391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:23.152303 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:23.874328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:23.874290 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:23.875092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:23.875073 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:27:24.172562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.172477 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7"] Nov 02 08:27:24.173010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.172997 5055 cpu_manager.go:414] "containerMap: removing container" podUID="39ed46ba-8dea-4868-a335-147ac8d8a683" containerName="revoke-keys" Nov 02 08:27:24.173052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.173011 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ed46ba-8dea-4868-a335-147ac8d8a683" containerName="revoke-keys" Nov 02 08:27:24.173165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.173153 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ed46ba-8dea-4868-a335-147ac8d8a683" containerName="revoke-keys" Nov 02 08:27:24.177101 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.177082 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.225671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.225638 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:27:24.225973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.225889 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:27:24.226948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.226921 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:27:24.227086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.226936 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:27:24.230128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.230105 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7"] Nov 02 08:27:24.260306 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.260270 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.260474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.260405 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.260610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.260522 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.362034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.362001 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.362273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.362068 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.362273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.362177 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.363059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.363033 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.365000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.364979 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.414928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.414894 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2\") pod \"maas-api-revoke-sub-f0387b5f-4hdd7\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.490631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.490571 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:24.876425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:24.876389 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7"] Nov 02 08:27:24.878789 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:24.878759 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45f3872_8609_4e15_b576_557114ef9ae8.slice/crio-74fa39c0db6af5f12f65b0b849a643c53ecda6252ddc1d7463f982e894bcda38 WatchSource:0}: Error finding container 74fa39c0db6af5f12f65b0b849a643c53ecda6252ddc1d7463f982e894bcda38: Status 404 returned error can't find the container with id 74fa39c0db6af5f12f65b0b849a643c53ecda6252ddc1d7463f982e894bcda38 Nov 02 08:27:24.942562 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:27:24.942525 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:27:25.891823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:25.891787 5055 generic.go:356] "Generic (PLEG): container finished" podID="e45f3872-8609-4e15-b576-557114ef9ae8" containerID="f5b367eda82e5cc62465d1422346ad2b195111e88f23eadd44ee7c4636ee5aa2" exitCode=0 Nov 02 08:27:25.892329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:25.891876 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" event={"ID":"e45f3872-8609-4e15-b576-557114ef9ae8","Type":"ContainerDied","Data":"f5b367eda82e5cc62465d1422346ad2b195111e88f23eadd44ee7c4636ee5aa2"} Nov 02 08:27:25.892329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:25.891926 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" event={"ID":"e45f3872-8609-4e15-b576-557114ef9ae8","Type":"ContainerStarted","Data":"74fa39c0db6af5f12f65b0b849a643c53ecda6252ddc1d7463f982e894bcda38"} Nov 02 08:27:27.079487 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.079459 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:27.199912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.199881 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2\") pod \"e45f3872-8609-4e15-b576-557114ef9ae8\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " Nov 02 08:27:27.200119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.200098 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca\") pod \"e45f3872-8609-4e15-b576-557114ef9ae8\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " Nov 02 08:27:27.200225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.200155 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca\") pod \"e45f3872-8609-4e15-b576-557114ef9ae8\" (UID: \"e45f3872-8609-4e15-b576-557114ef9ae8\") " Nov 02 08:27:27.201386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.201354 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca" pod "e45f3872-8609-4e15-b576-557114ef9ae8" (UID: "e45f3872-8609-4e15-b576-557114ef9ae8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:27.202748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.202720 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2" pod "e45f3872-8609-4e15-b576-557114ef9ae8" (UID: "e45f3872-8609-4e15-b576-557114ef9ae8"). InnerVolumeSpecName "kube-api-access-bhzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:27.202866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.202791 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca" pod "e45f3872-8609-4e15-b576-557114ef9ae8" (UID: "e45f3872-8609-4e15-b576-557114ef9ae8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:27.303103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.303066 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:27.303103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.303102 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f3872-8609-4e15-b576-557114ef9ae8-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:27.303103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.303112 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/e45f3872-8609-4e15-b576-557114ef9ae8-kube-api-access-bhzp2\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:27.910125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.910084 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" Nov 02 08:27:27.910125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.910097 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7" event={"ID":"e45f3872-8609-4e15-b576-557114ef9ae8","Type":"ContainerDied","Data":"74fa39c0db6af5f12f65b0b849a643c53ecda6252ddc1d7463f982e894bcda38"} Nov 02 08:27:27.910403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:27.910208 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74fa39c0db6af5f12f65b0b849a643c53ecda6252ddc1d7463f982e894bcda38" Nov 02 08:27:39.542795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.542714 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb"] Nov 02 08:27:39.543300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.543284 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e45f3872-8609-4e15-b576-557114ef9ae8" containerName="revoke-keys" Nov 02 08:27:39.543346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.543301 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45f3872-8609-4e15-b576-557114ef9ae8" containerName="revoke-keys" Nov 02 08:27:39.543428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.543419 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45f3872-8609-4e15-b576-557114ef9ae8" containerName="revoke-keys" Nov 02 08:27:39.547134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.547113 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.595980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.595946 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:27:39.641708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.641666 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:27:39.641708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.641704 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:27:39.641927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.641762 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:27:39.648108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.648074 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb"] Nov 02 08:27:39.673245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.673212 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.673404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.673360 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.673484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.673465 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm6hl\" (UniqueName: \"kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.775296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.775258 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.775474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.775352 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.775474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.775429 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm6hl\" (UniqueName: \"kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.776133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.776102 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.778072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.778045 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.828069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.827979 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm6hl\" (UniqueName: \"kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl\") pod \"maas-api-revoke-sub-e59cc315-dtzkb\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:39.861102 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:39.861068 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:40.238312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:40.238279 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb"] Nov 02 08:27:40.238450 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:40.238357 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fbaf00_98a2_48da_9b81_f5d1e723f021.slice/crio-a9285f15903f3d514f8a8caf6e4a298d2c281a1ab72a5d085699297de4c92848 WatchSource:0}: Error finding container a9285f15903f3d514f8a8caf6e4a298d2c281a1ab72a5d085699297de4c92848: Status 404 returned error can't find the container with id a9285f15903f3d514f8a8caf6e4a298d2c281a1ab72a5d085699297de4c92848 Nov 02 08:27:41.063154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:41.063093 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" event={"ID":"81fbaf00-98a2-48da-9b81-f5d1e723f021","Type":"ContainerStarted","Data":"43502cea273e146487c955ff00a34901ea388e18db8eff6f6839c1f4fec5162f"} Nov 02 08:27:41.063894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:41.063864 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" event={"ID":"81fbaf00-98a2-48da-9b81-f5d1e723f021","Type":"ContainerStarted","Data":"a9285f15903f3d514f8a8caf6e4a298d2c281a1ab72a5d085699297de4c92848"} Nov 02 08:27:41.171385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:41.171331 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" podStartSLOduration=2.171315701 podStartE2EDuration="2.171315701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:41.169524751 +0000 UTC m=+1308.260145169" watchObservedRunningTime="2026-11-02 08:27:41.171315701 +0000 UTC m=+1308.261936181" Nov 02 08:27:42.076485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:42.076444 5055 generic.go:356] "Generic (PLEG): container finished" podID="81fbaf00-98a2-48da-9b81-f5d1e723f021" containerID="43502cea273e146487c955ff00a34901ea388e18db8eff6f6839c1f4fec5162f" exitCode=0 Nov 02 08:27:42.077037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:42.076505 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" event={"ID":"81fbaf00-98a2-48da-9b81-f5d1e723f021","Type":"ContainerDied","Data":"43502cea273e146487c955ff00a34901ea388e18db8eff6f6839c1f4fec5162f"} Nov 02 08:27:43.263393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.263360 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:43.362349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.362198 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca\") pod \"81fbaf00-98a2-48da-9b81-f5d1e723f021\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " Nov 02 08:27:43.362349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.362335 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca\") pod \"81fbaf00-98a2-48da-9b81-f5d1e723f021\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " Nov 02 08:27:43.362637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.362572 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl\" (UniqueName: \"kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl\") pod \"81fbaf00-98a2-48da-9b81-f5d1e723f021\" (UID: \"81fbaf00-98a2-48da-9b81-f5d1e723f021\") " Nov 02 08:27:43.362778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.362737 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca" pod "81fbaf00-98a2-48da-9b81-f5d1e723f021" (UID: "81fbaf00-98a2-48da-9b81-f5d1e723f021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:43.363117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.363096 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:43.364915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.364884 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca" pod "81fbaf00-98a2-48da-9b81-f5d1e723f021" (UID: "81fbaf00-98a2-48da-9b81-f5d1e723f021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:43.365049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.364951 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl" pod "81fbaf00-98a2-48da-9b81-f5d1e723f021" (UID: "81fbaf00-98a2-48da-9b81-f5d1e723f021"). InnerVolumeSpecName "kube-api-access-wm6hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:43.464252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.464206 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm6hl\" (UniqueName: \"kubernetes.io/projected/81fbaf00-98a2-48da-9b81-f5d1e723f021-kube-api-access-wm6hl\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:43.464252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:43.464246 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fbaf00-98a2-48da-9b81-f5d1e723f021-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:44.096926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:44.096191 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" event={"ID":"81fbaf00-98a2-48da-9b81-f5d1e723f021","Type":"ContainerDied","Data":"a9285f15903f3d514f8a8caf6e4a298d2c281a1ab72a5d085699297de4c92848"} Nov 02 08:27:44.096926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:44.096305 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9285f15903f3d514f8a8caf6e4a298d2c281a1ab72a5d085699297de4c92848" Nov 02 08:27:44.096926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:44.096402 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb" Nov 02 08:27:49.436068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.436026 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9"] Nov 02 08:27:49.436852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.436653 5055 cpu_manager.go:414] "containerMap: removing container" podUID="81fbaf00-98a2-48da-9b81-f5d1e723f021" containerName="revoke-keys" Nov 02 08:27:49.436852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.436668 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fbaf00-98a2-48da-9b81-f5d1e723f021" containerName="revoke-keys" Nov 02 08:27:49.436852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.436815 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fbaf00-98a2-48da-9b81-f5d1e723f021" containerName="revoke-keys" Nov 02 08:27:49.440647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.440623 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.537015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.536365 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:27:49.537015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.536382 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:27:49.537015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.536382 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:27:49.537015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.536946 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:27:49.542194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.542164 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9"] Nov 02 08:27:49.550741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.550714 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.550876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.550778 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.550919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.550879 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptff\" (UniqueName: \"kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.652409 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.652373 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.652590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.652445 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.652590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.652496 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptff\" (UniqueName: \"kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.653230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.653178 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.658440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.657933 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.706131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.706098 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptff\" (UniqueName: \"kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff\") pod \"maas-api-revoke-sub-cbe4b070-d4wv9\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:49.754713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:49.754675 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:50.129804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:50.129777 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9"] Nov 02 08:27:50.131007 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:50.130978 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc44d36_68c6_47f0_b967_6147017bc5b0.slice/crio-4e966421114e4c7a8e4d88978c60ce66eb23c1ec2aca4fd77ea033b2e3520088 WatchSource:0}: Error finding container 4e966421114e4c7a8e4d88978c60ce66eb23c1ec2aca4fd77ea033b2e3520088: Status 404 returned error can't find the container with id 4e966421114e4c7a8e4d88978c60ce66eb23c1ec2aca4fd77ea033b2e3520088 Nov 02 08:27:50.145474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:50.145439 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" event={"ID":"6bc44d36-68c6-47f0-b967-6147017bc5b0","Type":"ContainerStarted","Data":"4e966421114e4c7a8e4d88978c60ce66eb23c1ec2aca4fd77ea033b2e3520088"} Nov 02 08:27:51.153569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:51.153536 5055 generic.go:356] "Generic (PLEG): container finished" podID="6bc44d36-68c6-47f0-b967-6147017bc5b0" containerID="8dbd01d34fcfb1c33ce7e610501deff635976bd93111d842b0b2422a0e27bac1" exitCode=0 Nov 02 08:27:51.154027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:51.153621 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" event={"ID":"6bc44d36-68c6-47f0-b967-6147017bc5b0","Type":"ContainerDied","Data":"8dbd01d34fcfb1c33ce7e610501deff635976bd93111d842b0b2422a0e27bac1"} Nov 02 08:27:52.332646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.332618 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:52.389597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.389566 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca\") pod \"6bc44d36-68c6-47f0-b967-6147017bc5b0\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " Nov 02 08:27:52.389789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.389608 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca\") pod \"6bc44d36-68c6-47f0-b967-6147017bc5b0\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " Nov 02 08:27:52.389789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.389734 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff\" (UniqueName: \"kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff\") pod \"6bc44d36-68c6-47f0-b967-6147017bc5b0\" (UID: \"6bc44d36-68c6-47f0-b967-6147017bc5b0\") " Nov 02 08:27:52.390032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.390008 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca" pod "6bc44d36-68c6-47f0-b967-6147017bc5b0" (UID: "6bc44d36-68c6-47f0-b967-6147017bc5b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:52.391687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.391665 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca" pod "6bc44d36-68c6-47f0-b967-6147017bc5b0" (UID: "6bc44d36-68c6-47f0-b967-6147017bc5b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:52.391837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.391820 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff" pod "6bc44d36-68c6-47f0-b967-6147017bc5b0" (UID: "6bc44d36-68c6-47f0-b967-6147017bc5b0"). InnerVolumeSpecName "kube-api-access-8ptff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:52.491852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.491818 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:52.491852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.491850 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc44d36-68c6-47f0-b967-6147017bc5b0-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:52.491852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:52.491860 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptff\" (UniqueName: \"kubernetes.io/projected/6bc44d36-68c6-47f0-b967-6147017bc5b0-kube-api-access-8ptff\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:53.171239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.171190 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" event={"ID":"6bc44d36-68c6-47f0-b967-6147017bc5b0","Type":"ContainerDied","Data":"4e966421114e4c7a8e4d88978c60ce66eb23c1ec2aca4fd77ea033b2e3520088"} Nov 02 08:27:53.171239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.171217 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9" Nov 02 08:27:53.171427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.171256 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e966421114e4c7a8e4d88978c60ce66eb23c1ec2aca4fd77ea033b2e3520088" Nov 02 08:27:53.427589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.427505 5055 scope.go:122] "RemoveContainer" containerID="d13a621be2589f53809cac8c95abbf8e165a54823c99abc62c0eee12dc03c8c9" Nov 02 08:27:53.439527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.439500 5055 scope.go:122] "RemoveContainer" containerID="2ddba2846ce0843099c03b5595beb72401b536a03c885f91201ac58b6a2462fb" Nov 02 08:27:53.449579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.449557 5055 scope.go:122] "RemoveContainer" containerID="d3a50c87a9fc84f0f3b69ddda579e2e03f63260d5a4cf230aed94b5f751742c6" Nov 02 08:27:53.459085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:53.459060 5055 scope.go:122] "RemoveContainer" containerID="fe50d9ab80ac99eabf97004e59a0e3a4b5963470a219c37e195271ef93f1cf88" Nov 02 08:27:54.921977 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:27:54.921934 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:27:56.769790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.769757 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh"] Nov 02 08:27:56.770353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.770336 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6bc44d36-68c6-47f0-b967-6147017bc5b0" containerName="revoke-keys" Nov 02 08:27:56.770405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.770356 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc44d36-68c6-47f0-b967-6147017bc5b0" containerName="revoke-keys" Nov 02 08:27:56.770500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.770488 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc44d36-68c6-47f0-b967-6147017bc5b0" containerName="revoke-keys" Nov 02 08:27:56.775643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.775624 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.824586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.824556 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:27:56.845598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.844057 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh"] Nov 02 08:27:56.847449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.847416 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.847599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.847493 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfzp\" (UniqueName: \"kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.847669 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.847605 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.870335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.869076 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:27:56.870335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.869085 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:27:56.870629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.870507 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:27:56.949188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.949131 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.949377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.949300 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.949377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.949354 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfzp\" (UniqueName: \"kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.950000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.949967 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:56.951607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:56.951587 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:57.003762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:57.003722 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfzp\" (UniqueName: \"kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp\") pod \"maas-api-revoke-sub-d53f6fd3-2rrvh\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:57.095312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:57.095239 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:27:57.475239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:27:57.475206 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed9885e_9892_479b_838f_f3981a0271d7.slice/crio-93d43dcfcc0eba960aa799a8a8eef7dd5b3cfa4ed1b82aa0a1a457e66a9ce731 WatchSource:0}: Error finding container 93d43dcfcc0eba960aa799a8a8eef7dd5b3cfa4ed1b82aa0a1a457e66a9ce731: Status 404 returned error can't find the container with id 93d43dcfcc0eba960aa799a8a8eef7dd5b3cfa4ed1b82aa0a1a457e66a9ce731 Nov 02 08:27:57.476759 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:57.476719 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh"] Nov 02 08:27:58.212916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:58.212874 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" event={"ID":"6ed9885e-9892-479b-838f-f3981a0271d7","Type":"ContainerStarted","Data":"805f9d6a7d25d8084b786e6476cfd5a049ee7bcd758d1df37f22e3e03dea405c"} Nov 02 08:27:58.213350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:58.212940 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" event={"ID":"6ed9885e-9892-479b-838f-f3981a0271d7","Type":"ContainerStarted","Data":"93d43dcfcc0eba960aa799a8a8eef7dd5b3cfa4ed1b82aa0a1a457e66a9ce731"} Nov 02 08:27:58.328341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:58.328280 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" podStartSLOduration=2.328260793 podStartE2EDuration="2.328260793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:58.323928754 +0000 UTC m=+1325.414549173" watchObservedRunningTime="2026-11-02 08:27:58.328260793 +0000 UTC m=+1325.418881213" Nov 02 08:27:59.222609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:59.222568 5055 generic.go:356] "Generic (PLEG): container finished" podID="6ed9885e-9892-479b-838f-f3981a0271d7" containerID="805f9d6a7d25d8084b786e6476cfd5a049ee7bcd758d1df37f22e3e03dea405c" exitCode=0 Nov 02 08:27:59.223020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:27:59.222650 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" event={"ID":"6ed9885e-9892-479b-838f-f3981a0271d7","Type":"ContainerDied","Data":"805f9d6a7d25d8084b786e6476cfd5a049ee7bcd758d1df37f22e3e03dea405c"} Nov 02 08:28:00.185811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.185771 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6"] Nov 02 08:28:00.190193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.190166 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.248740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.248693 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6"] Nov 02 08:28:00.299162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.291294 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zh8\" (UniqueName: \"kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.299162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.291485 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.299162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.291651 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.392669 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.392629 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zh8\" (UniqueName: \"kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.392919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.392745 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.392919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.392897 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.393600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.393568 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.395560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.395529 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.428860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.428831 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:28:00.446730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.446659 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zh8\" (UniqueName: \"kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8\") pod \"maas-api-revoke-sub-c5219343-rcln6\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.494352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.494318 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca\") pod \"6ed9885e-9892-479b-838f-f3981a0271d7\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " Nov 02 08:28:00.494352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.494357 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp\" (UniqueName: \"kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp\") pod \"6ed9885e-9892-479b-838f-f3981a0271d7\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " Nov 02 08:28:00.494675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.494434 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca\") pod \"6ed9885e-9892-479b-838f-f3981a0271d7\" (UID: \"6ed9885e-9892-479b-838f-f3981a0271d7\") " Nov 02 08:28:00.494833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.494808 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca" pod "6ed9885e-9892-479b-838f-f3981a0271d7" (UID: "6ed9885e-9892-479b-838f-f3981a0271d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:00.495079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.495059 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:00.496656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.496632 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca" pod "6ed9885e-9892-479b-838f-f3981a0271d7" (UID: "6ed9885e-9892-479b-838f-f3981a0271d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:00.497049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.497021 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp" pod "6ed9885e-9892-479b-838f-f3981a0271d7" (UID: "6ed9885e-9892-479b-838f-f3981a0271d7"). InnerVolumeSpecName "kube-api-access-vdfzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:00.506388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.506358 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:00.596068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.596041 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed9885e-9892-479b-838f-f3981a0271d7-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:00.597196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.597178 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfzp\" (UniqueName: \"kubernetes.io/projected/6ed9885e-9892-479b-838f-f3981a0271d7-kube-api-access-vdfzp\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:00.894318 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:00.894275 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2843f742_c60d_47ed_a8e8_e2553f858261.slice/crio-5580d850a17a6d19e94a2eaaaa4cf81988ce2a6d0e49abcd664bed07b7401e1d WatchSource:0}: Error finding container 5580d850a17a6d19e94a2eaaaa4cf81988ce2a6d0e49abcd664bed07b7401e1d: Status 404 returned error can't find the container with id 5580d850a17a6d19e94a2eaaaa4cf81988ce2a6d0e49abcd664bed07b7401e1d Nov 02 08:28:00.895128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:00.895097 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6"] Nov 02 08:28:01.242615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:01.242587 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" Nov 02 08:28:01.242767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:01.242626 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh" event={"ID":"6ed9885e-9892-479b-838f-f3981a0271d7","Type":"ContainerDied","Data":"93d43dcfcc0eba960aa799a8a8eef7dd5b3cfa4ed1b82aa0a1a457e66a9ce731"} Nov 02 08:28:01.242767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:01.242682 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93d43dcfcc0eba960aa799a8a8eef7dd5b3cfa4ed1b82aa0a1a457e66a9ce731" Nov 02 08:28:01.244476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:01.244449 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" event={"ID":"2843f742-c60d-47ed-a8e8-e2553f858261","Type":"ContainerStarted","Data":"370b0a815b1765e5847ec558ce2e04f64236f1e7a45313540e4ff9e68d4411df"} Nov 02 08:28:01.245060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:01.244489 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" event={"ID":"2843f742-c60d-47ed-a8e8-e2553f858261","Type":"ContainerStarted","Data":"5580d850a17a6d19e94a2eaaaa4cf81988ce2a6d0e49abcd664bed07b7401e1d"} Nov 02 08:28:01.353424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:01.353372 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" podStartSLOduration=1.353354825 podStartE2EDuration="1.353354825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:01.3508027 +0000 UTC m=+1328.441423120" watchObservedRunningTime="2026-11-02 08:28:01.353354825 +0000 UTC m=+1328.443975259" Nov 02 08:28:02.256241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:02.256125 5055 generic.go:356] "Generic (PLEG): container finished" podID="2843f742-c60d-47ed-a8e8-e2553f858261" containerID="370b0a815b1765e5847ec558ce2e04f64236f1e7a45313540e4ff9e68d4411df" exitCode=0 Nov 02 08:28:02.256445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:02.256362 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" event={"ID":"2843f742-c60d-47ed-a8e8-e2553f858261","Type":"ContainerDied","Data":"370b0a815b1765e5847ec558ce2e04f64236f1e7a45313540e4ff9e68d4411df"} Nov 02 08:28:03.448932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.448907 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:03.466826 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.466793 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv"] Nov 02 08:28:03.467398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.467384 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6ed9885e-9892-479b-838f-f3981a0271d7" containerName="revoke-keys" Nov 02 08:28:03.467458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.467399 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed9885e-9892-479b-838f-f3981a0271d7" containerName="revoke-keys" Nov 02 08:28:03.467458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.467414 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2843f742-c60d-47ed-a8e8-e2553f858261" containerName="revoke-keys" Nov 02 08:28:03.467458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.467420 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2843f742-c60d-47ed-a8e8-e2553f858261" containerName="revoke-keys" Nov 02 08:28:03.467554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.467507 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed9885e-9892-479b-838f-f3981a0271d7" containerName="revoke-keys" Nov 02 08:28:03.467554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.467516 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2843f742-c60d-47ed-a8e8-e2553f858261" containerName="revoke-keys" Nov 02 08:28:03.470909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.470893 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.520349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.520253 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv"] Nov 02 08:28:03.543020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.542985 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca\") pod \"2843f742-c60d-47ed-a8e8-e2553f858261\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " Nov 02 08:28:03.543229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543083 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8\" (UniqueName: \"kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8\") pod \"2843f742-c60d-47ed-a8e8-e2553f858261\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " Nov 02 08:28:03.543229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543112 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca\") pod \"2843f742-c60d-47ed-a8e8-e2553f858261\" (UID: \"2843f742-c60d-47ed-a8e8-e2553f858261\") " Nov 02 08:28:03.543506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543485 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.543608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543562 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca" pod "2843f742-c60d-47ed-a8e8-e2553f858261" (UID: "2843f742-c60d-47ed-a8e8-e2553f858261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:03.543671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543600 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.543671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543651 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.543747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543720 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.543868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.543848 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jw5g\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.544052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.544032 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.544130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.544114 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.544279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.544261 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.544331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.544311 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.544478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.544457 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:03.545512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.545488 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8" pod "2843f742-c60d-47ed-a8e8-e2553f858261" (UID: "2843f742-c60d-47ed-a8e8-e2553f858261"). InnerVolumeSpecName "kube-api-access-b8zh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:03.545819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.545801 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca" pod "2843f742-c60d-47ed-a8e8-e2553f858261" (UID: "2843f742-c60d-47ed-a8e8-e2553f858261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:03.645321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645280 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jw5g\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645409 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645473 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645540 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645581 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645649 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645682 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645712 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645751 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.645897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645853 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f742-c60d-47ed-a8e8-e2553f858261-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:03.645897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.645872 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zh8\" (UniqueName: \"kubernetes.io/projected/2843f742-c60d-47ed-a8e8-e2553f858261-kube-api-access-b8zh8\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:03.646212 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.646124 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.646323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.646205 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.646323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.646250 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.646323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.646283 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.646437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.646399 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.647960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.647936 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.648215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.648197 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.698962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.698924 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.699532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.699504 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jw5g\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:03.785264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:03.785172 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:04.222970 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:04.222935 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fb13e2_d141_4e0c_ab62_371193b96d49.slice/crio-2a40fe6dd2c20215d2aa5fdb5ba67f8c28eefe4ee3e3ee8306093427bf2d1b21 WatchSource:0}: Error finding container 2a40fe6dd2c20215d2aa5fdb5ba67f8c28eefe4ee3e3ee8306093427bf2d1b21: Status 404 returned error can't find the container with id 2a40fe6dd2c20215d2aa5fdb5ba67f8c28eefe4ee3e3ee8306093427bf2d1b21 Nov 02 08:28:04.224173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.224115 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv"] Nov 02 08:28:04.225969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.225931 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:04.226088 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.226021 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:04.226088 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.226077 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:04.280000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.279961 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" event={"ID":"12fb13e2-d141-4e0c-ab62-371193b96d49","Type":"ContainerStarted","Data":"2a40fe6dd2c20215d2aa5fdb5ba67f8c28eefe4ee3e3ee8306093427bf2d1b21"} Nov 02 08:28:04.281780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.281753 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" event={"ID":"2843f742-c60d-47ed-a8e8-e2553f858261","Type":"ContainerDied","Data":"5580d850a17a6d19e94a2eaaaa4cf81988ce2a6d0e49abcd664bed07b7401e1d"} Nov 02 08:28:04.281780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.281769 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6" Nov 02 08:28:04.281924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:04.281803 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5580d850a17a6d19e94a2eaaaa4cf81988ce2a6d0e49abcd664bed07b7401e1d" Nov 02 08:28:05.323365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:05.323325 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" event={"ID":"12fb13e2-d141-4e0c-ab62-371193b96d49","Type":"ContainerStarted","Data":"1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01"} Nov 02 08:28:05.517205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:05.506333 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podStartSLOduration=2.506306796 podStartE2EDuration="2.506306796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:05.478833709 +0000 UTC m=+1332.569454128" watchObservedRunningTime="2026-11-02 08:28:05.506306796 +0000 UTC m=+1332.596927217" Nov 02 08:28:05.803176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:05.802549 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:05.812193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:05.811297 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:05.812193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:05.811355 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:06.791556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:06.791514 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:06.792250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:06.792216 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:07.796642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:07.796598 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:07.797277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:07.797239 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:07.948342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:07.948296 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:28:08.791166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:08.790589 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:08.791166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:08.790653 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:09.807326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:09.806305 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:09.807326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:09.806364 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:10.186235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.182209 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:10.208168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.207755 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 02 08:28:10.341288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.339489 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 08:28:10.388167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.387382 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 08:28:10.518190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.517528 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:10.630164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.619238 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:10.730179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.708364 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 08:28:10.815514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.815393 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:10.820909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.820797 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:10.942164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:10.924272 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:11.219164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.215234 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:11.219164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.215944 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:28:11.298166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.295232 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:11.340673 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.340627 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:11.341331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.341303 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:28:11.405165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.388878 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" podUID="5d13bf2a-d174-4b83-b363-ed42f963cff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 02 08:28:11.405165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.389299 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 08:28:11.405165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.393222 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:28:11.588084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.506112 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 02 08:28:11.588691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.568285 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 08:28:11.590027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.589999 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:28:11.802173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.791543 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 08:28:11.802173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.791900 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 08:28:11.841219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.802574 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:28:11.906172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.845240 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:11.906172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.849222 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 08:28:11.906172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.874559 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 08:28:11.906172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.874653 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 08:28:11.906172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.878625 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Nov 02 08:28:11.906172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.878823 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:28:11.945164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.939538 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:11.968169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:11.958926 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:28:12.184794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.184710 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:28:12.279167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.249901 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:28:12.296218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.296168 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:12.327161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.321692 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:12.366533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.366502 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:28:12.366830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.366815 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:28:12.367007 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.366995 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:28:12.367190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.367177 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:28:12.429395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.429362 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:28:12.587758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.587727 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:28:12.809723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:12.809693 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:28:13.789659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:13.789620 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:28:13.790216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:13.789712 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:14.833361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:14.833309 5055 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:28:14.833893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:14.833394 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:28:15.123307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:15.122598 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:15.123307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:15.123260 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:15.124908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:15.124868 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:28:16.938251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:16.938212 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4"] Nov 02 08:28:16.943225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:16.943204 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:16.992866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:16.992838 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:28:17.038820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.038763 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:28:17.038820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.038803 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:28:17.039087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.038889 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:28:17.043324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.043298 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4"] Nov 02 08:28:17.061838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.061808 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.061959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.061925 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqfc\" (UniqueName: \"kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.062000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.061971 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.163908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.163866 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.164349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.164326 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqfc\" (UniqueName: \"kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.164506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.164490 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.165777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.165745 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.168367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.168309 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.219290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.219238 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqfc\" (UniqueName: \"kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc\") pod \"maas-api-revoke-sub-b835a782-tftc4\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.257340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.257300 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:17.641875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.641835 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4"] Nov 02 08:28:17.643300 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:17.643269 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b14dd0_939b_4de2_9465_c655ac236d3f.slice/crio-eb52f8dd6a477db0b21e7849e9bc4df50f5045b9dad109e9ea99ce00c75d7b50 WatchSource:0}: Error finding container eb52f8dd6a477db0b21e7849e9bc4df50f5045b9dad109e9ea99ce00c75d7b50: Status 404 returned error can't find the container with id eb52f8dd6a477db0b21e7849e9bc4df50f5045b9dad109e9ea99ce00c75d7b50 Nov 02 08:28:17.857225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:17.857178 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" event={"ID":"68b14dd0-939b-4de2-9465-c655ac236d3f","Type":"ContainerStarted","Data":"eb52f8dd6a477db0b21e7849e9bc4df50f5045b9dad109e9ea99ce00c75d7b50"} Nov 02 08:28:18.866451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:18.866412 5055 generic.go:356] "Generic (PLEG): container finished" podID="68b14dd0-939b-4de2-9465-c655ac236d3f" containerID="c1982f5d36507c40edeb01ba3c848a38358879f650f6a9a19edb9e72322c21fb" exitCode=0 Nov 02 08:28:18.866858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:18.866499 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" event={"ID":"68b14dd0-939b-4de2-9465-c655ac236d3f","Type":"ContainerDied","Data":"c1982f5d36507c40edeb01ba3c848a38358879f650f6a9a19edb9e72322c21fb"} Nov 02 08:28:19.055451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.055410 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk"] Nov 02 08:28:19.059346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.059312 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606d263c-zj7rk"] Nov 02 08:28:19.093875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.093845 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39837395-1069-418c-a113-142d1cd57901" path="/var/lib/kubelet/pods/39837395-1069-418c-a113-142d1cd57901/volumes" Nov 02 08:28:19.276661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.276627 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s"] Nov 02 08:28:19.281123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.281100 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.373122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.373031 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f5bb4f-openshift-default-dockercfg-p9lbg\"" Nov 02 08:28:19.395399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.395364 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s"] Nov 02 08:28:19.401935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.396962 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.402262 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.402239 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.402608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.402474 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.443961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.438348 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtx2\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.443961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.438421 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.443961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.438537 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.443961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.438605 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.443961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.438723 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.443961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.438771 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.540743 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.540829 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.540974 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541046 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtx2\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541085 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541198 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541260 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541363 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541407 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541831 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.541934 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.542038 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.542078 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.547340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.542958 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.548056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.547974 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.551187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.550733 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.595059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.595025 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.595244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.595212 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtx2\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2\") pod \"e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:19.595455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:19.595440 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:20.020619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.020585 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:20.025622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.025593 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s"] Nov 02 08:28:20.025983 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:20.025958 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765bdaab_c78e_4491_83f1_2c0bc4f4d510.slice/crio-3f060d55671c4d3a59191bcabe1fb0c1b704f6b31782acbcb7ded11d00645365 WatchSource:0}: Error finding container 3f060d55671c4d3a59191bcabe1fb0c1b704f6b31782acbcb7ded11d00645365: Status 404 returned error can't find the container with id 3f060d55671c4d3a59191bcabe1fb0c1b704f6b31782acbcb7ded11d00645365 Nov 02 08:28:20.028417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.028386 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:20.028486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.028446 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:20.028486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.028472 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:20.153158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.152408 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc\" (UniqueName: \"kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc\") pod \"68b14dd0-939b-4de2-9465-c655ac236d3f\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " Nov 02 08:28:20.153158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.152618 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca\") pod \"68b14dd0-939b-4de2-9465-c655ac236d3f\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " Nov 02 08:28:20.153158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.152703 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca\") pod \"68b14dd0-939b-4de2-9465-c655ac236d3f\" (UID: \"68b14dd0-939b-4de2-9465-c655ac236d3f\") " Nov 02 08:28:20.157631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.157342 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca" pod "68b14dd0-939b-4de2-9465-c655ac236d3f" (UID: "68b14dd0-939b-4de2-9465-c655ac236d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:20.163850 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.163819 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc" pod "68b14dd0-939b-4de2-9465-c655ac236d3f" (UID: "68b14dd0-939b-4de2-9465-c655ac236d3f"). InnerVolumeSpecName "kube-api-access-6xqfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:20.165549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.165515 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca" pod "68b14dd0-939b-4de2-9465-c655ac236d3f" (UID: "68b14dd0-939b-4de2-9465-c655ac236d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:20.254493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.254461 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:20.254493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.254491 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b14dd0-939b-4de2-9465-c655ac236d3f-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:20.254493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.254501 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqfc\" (UniqueName: \"kubernetes.io/projected/68b14dd0-939b-4de2-9465-c655ac236d3f-kube-api-access-6xqfc\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:20.881281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.881196 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" Nov 02 08:28:20.881281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.881252 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4" event={"ID":"68b14dd0-939b-4de2-9465-c655ac236d3f","Type":"ContainerDied","Data":"eb52f8dd6a477db0b21e7849e9bc4df50f5045b9dad109e9ea99ce00c75d7b50"} Nov 02 08:28:20.881511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.881328 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb52f8dd6a477db0b21e7849e9bc4df50f5045b9dad109e9ea99ce00c75d7b50" Nov 02 08:28:20.883131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.883102 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" event={"ID":"765bdaab-c78e-4491-83f1-2c0bc4f4d510","Type":"ContainerStarted","Data":"24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59"} Nov 02 08:28:20.883292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:20.883175 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" event={"ID":"765bdaab-c78e-4491-83f1-2c0bc4f4d510","Type":"ContainerStarted","Data":"3f060d55671c4d3a59191bcabe1fb0c1b704f6b31782acbcb7ded11d00645365"} Nov 02 08:28:21.040123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:21.040068 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podStartSLOduration=2.040052846 podStartE2EDuration="2.040052846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:20.994086179 +0000 UTC m=+1348.084706589" watchObservedRunningTime="2026-11-02 08:28:21.040052846 +0000 UTC m=+1348.130673265" Nov 02 08:28:21.595926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:21.595892 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:21.602121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:21.602094 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:21.892556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:21.892468 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:21.893417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:21.893394 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:28:24.925261 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:28:24.925211 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:28:29.817496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.817457 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874"] Nov 02 08:28:29.818430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.818400 5055 cpu_manager.go:414] "containerMap: removing container" podUID="68b14dd0-939b-4de2-9465-c655ac236d3f" containerName="revoke-keys" Nov 02 08:28:29.818430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.818424 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b14dd0-939b-4de2-9465-c655ac236d3f" containerName="revoke-keys" Nov 02 08:28:29.818586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.818574 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b14dd0-939b-4de2-9465-c655ac236d3f" containerName="revoke-keys" Nov 02 08:28:29.824066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.824047 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.978587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.978557 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b26f03-openshift-default-dockercfg-whtvz\"" Nov 02 08:28:29.988597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.988567 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874"] Nov 02 08:28:29.999485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999451 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999503 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999641 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999705 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999766 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999815 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7dz\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999934 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999873 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999934 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999899 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:29.999934 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:29.999929 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101262 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101350 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101395 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7dz\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101452 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101476 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101506 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101609 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.101758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101636 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.102068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101787 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.102068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.101796 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.102508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.102480 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.102759 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.102737 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.102863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.102836 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.103113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.103090 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.104711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.104686 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.106387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.106355 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.157122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.157085 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.157291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.157174 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7dz\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz\") pod \"e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.443449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.443349 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:30.876923 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:30.876884 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafdd2c8_572f_4ff1_ade7_431fb9d39a9a.slice/crio-18bf9da9d016c5ccb45a32c7aa770fce1903474980ab7c3df9fe29d47e9aa409 WatchSource:0}: Error finding container 18bf9da9d016c5ccb45a32c7aa770fce1903474980ab7c3df9fe29d47e9aa409: Status 404 returned error can't find the container with id 18bf9da9d016c5ccb45a32c7aa770fce1903474980ab7c3df9fe29d47e9aa409 Nov 02 08:28:30.877295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.876939 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874"] Nov 02 08:28:30.879399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.879363 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:30.879527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.879437 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:30.879527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:30.879483 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:31.004014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:31.003976 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" event={"ID":"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a","Type":"ContainerStarted","Data":"717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c"} Nov 02 08:28:31.004224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:31.004042 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" event={"ID":"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a","Type":"ContainerStarted","Data":"18bf9da9d016c5ccb45a32c7aa770fce1903474980ab7c3df9fe29d47e9aa409"} Nov 02 08:28:31.443733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:31.443697 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:31.449817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:31.449784 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:31.561539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:31.561482 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podStartSLOduration=2.561464413 podStartE2EDuration="2.561464413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:31.118408655 +0000 UTC m=+1358.209029096" watchObservedRunningTime="2026-11-02 08:28:31.561464413 +0000 UTC m=+1358.652084832" Nov 02 08:28:32.013396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:32.013365 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:32.014597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:32.014576 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:28:41.162588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.162550 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5"] Nov 02 08:28:41.166907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.166883 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.183817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.183785 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.183976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.183849 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7bs\" (UniqueName: \"kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.183976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.183886 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.219135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.219100 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:28:41.225992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.225949 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5"] Nov 02 08:28:41.226987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.226957 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:28:41.227301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.227267 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:28:41.265060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.265026 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:28:41.289095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.289048 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.289316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.289197 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7bs\" (UniqueName: \"kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.289316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.289264 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.290107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.290083 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.292381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.292358 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.344932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.344901 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7bs\" (UniqueName: \"kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs\") pod \"maas-api-revoke-sub-a5ebfca5-ghvd5\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.486050 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.486006 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:41.877682 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:41.877647 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08548a78_25ad_4f26_89af_cd304ca47a67.slice/crio-4233a1077ce51774f3b6804a3aed120578c47843a5349926915fbf4faa0eae0e WatchSource:0}: Error finding container 4233a1077ce51774f3b6804a3aed120578c47843a5349926915fbf4faa0eae0e: Status 404 returned error can't find the container with id 4233a1077ce51774f3b6804a3aed120578c47843a5349926915fbf4faa0eae0e Nov 02 08:28:41.878370 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:41.878340 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5"] Nov 02 08:28:42.444433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:42.444209 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" event={"ID":"08548a78-25ad-4f26-89af-cd304ca47a67","Type":"ContainerStarted","Data":"8a50c28495bec7a014c7d409070bb38b8dead75f851be0b5667bf4cf28b31b96"} Nov 02 08:28:42.444433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:42.444272 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" event={"ID":"08548a78-25ad-4f26-89af-cd304ca47a67","Type":"ContainerStarted","Data":"4233a1077ce51774f3b6804a3aed120578c47843a5349926915fbf4faa0eae0e"} Nov 02 08:28:42.554632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:42.554524 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" podStartSLOduration=1.554508964 podStartE2EDuration="1.554508964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:42.552538519 +0000 UTC m=+1369.643158935" watchObservedRunningTime="2026-11-02 08:28:42.554508964 +0000 UTC m=+1369.645129411" Nov 02 08:28:43.453918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:43.453884 5055 generic.go:356] "Generic (PLEG): container finished" podID="08548a78-25ad-4f26-89af-cd304ca47a67" containerID="8a50c28495bec7a014c7d409070bb38b8dead75f851be0b5667bf4cf28b31b96" exitCode=0 Nov 02 08:28:43.454381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:43.453962 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" event={"ID":"08548a78-25ad-4f26-89af-cd304ca47a67","Type":"ContainerDied","Data":"8a50c28495bec7a014c7d409070bb38b8dead75f851be0b5667bf4cf28b31b96"} Nov 02 08:28:44.219763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.219727 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8"] Nov 02 08:28:44.225124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.225102 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.289500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.289464 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8"] Nov 02 08:28:44.339738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.339693 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340014 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340240 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmth9\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340442 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340502 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340606 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340684 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340807 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.340889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.340854 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442175 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442228 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442280 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmth9\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442475 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442540 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442592 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442578 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442644 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442713 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442643 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442741 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442853 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.442964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.443020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.442971 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.443334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.443311 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.444989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.444961 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.445221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.445200 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.502707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.502631 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmth9\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.506537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.506479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.544382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.544332 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:44.662332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.661087 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:44.746980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.746936 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca\") pod \"08548a78-25ad-4f26-89af-cd304ca47a67\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " Nov 02 08:28:44.747211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.747090 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs\" (UniqueName: \"kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs\") pod \"08548a78-25ad-4f26-89af-cd304ca47a67\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " Nov 02 08:28:44.747211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.747132 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca\") pod \"08548a78-25ad-4f26-89af-cd304ca47a67\" (UID: \"08548a78-25ad-4f26-89af-cd304ca47a67\") " Nov 02 08:28:44.747893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.747864 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca" pod "08548a78-25ad-4f26-89af-cd304ca47a67" (UID: "08548a78-25ad-4f26-89af-cd304ca47a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:44.749671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.749638 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca" pod "08548a78-25ad-4f26-89af-cd304ca47a67" (UID: "08548a78-25ad-4f26-89af-cd304ca47a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:44.750114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.750087 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs" pod "08548a78-25ad-4f26-89af-cd304ca47a67" (UID: "08548a78-25ad-4f26-89af-cd304ca47a67"). InnerVolumeSpecName "kube-api-access-6k7bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:44.849667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.849589 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k7bs\" (UniqueName: \"kubernetes.io/projected/08548a78-25ad-4f26-89af-cd304ca47a67-kube-api-access-6k7bs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:44.849667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.849623 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:44.849667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:44.849635 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08548a78-25ad-4f26-89af-cd304ca47a67-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:45.010203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.010164 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8"] Nov 02 08:28:45.012239 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:45.012203 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dca4a6_8732_4d46_970c_f5db40b95df9.slice/crio-336e01b4ec32c65ac5a2d82d493a881fc5ccf07a120e3fa962847a4c91d31744 WatchSource:0}: Error finding container 336e01b4ec32c65ac5a2d82d493a881fc5ccf07a120e3fa962847a4c91d31744: Status 404 returned error can't find the container with id 336e01b4ec32c65ac5a2d82d493a881fc5ccf07a120e3fa962847a4c91d31744 Nov 02 08:28:45.015381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.015345 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:45.015485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.015414 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:45.015485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.015443 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:45.473406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.473300 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" event={"ID":"b4dca4a6-8732-4d46-970c-f5db40b95df9","Type":"ContainerStarted","Data":"9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9"} Nov 02 08:28:45.473406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.473371 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" event={"ID":"b4dca4a6-8732-4d46-970c-f5db40b95df9","Type":"ContainerStarted","Data":"336e01b4ec32c65ac5a2d82d493a881fc5ccf07a120e3fa962847a4c91d31744"} Nov 02 08:28:45.475301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.475277 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" Nov 02 08:28:45.475301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.475281 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5" event={"ID":"08548a78-25ad-4f26-89af-cd304ca47a67","Type":"ContainerDied","Data":"4233a1077ce51774f3b6804a3aed120578c47843a5349926915fbf4faa0eae0e"} Nov 02 08:28:45.475465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.475334 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4233a1077ce51774f3b6804a3aed120578c47843a5349926915fbf4faa0eae0e" Nov 02 08:28:45.545149 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.545098 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:45.583591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:45.583531 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podStartSLOduration=1.583516344 podStartE2EDuration="1.583516344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:45.582811189 +0000 UTC m=+1372.673431608" watchObservedRunningTime="2026-11-02 08:28:45.583516344 +0000 UTC m=+1372.674136779" Nov 02 08:28:46.492553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.492511 5055 generic.go:356] "Generic (PLEG): container finished" podID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerID="eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3" exitCode=0 Nov 02 08:28:46.494019 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.493986 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" event={"ID":"0dede8b3-0d74-4932-9aa0-248f41ab23ea","Type":"ContainerDied","Data":"eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3"} Nov 02 08:28:46.494664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.494641 5055 scope.go:122] "RemoveContainer" containerID="eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3" Nov 02 08:28:46.509182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.506193 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:46.509182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.506312 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:46.509182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.506368 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:28:46.546886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.546851 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:46.547400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:46.546915 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:47.546163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:47.545388 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:47.546163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:47.545491 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:47.553227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:47.547342 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" event={"ID":"0dede8b3-0d74-4932-9aa0-248f41ab23ea","Type":"ContainerStarted","Data":"ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c"} Nov 02 08:28:48.157305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.157203 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:28:48.161969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.160519 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:28:48.161969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.160621 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:28:48.546075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.546020 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:48.546302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.546105 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:48.848300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.844359 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7"] Nov 02 08:28:48.848300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.845324 5055 cpu_manager.go:414] "containerMap: removing container" podUID="08548a78-25ad-4f26-89af-cd304ca47a67" containerName="revoke-keys" Nov 02 08:28:48.848300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.845345 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08548a78-25ad-4f26-89af-cd304ca47a67" containerName="revoke-keys" Nov 02 08:28:48.848300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.845538 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="08548a78-25ad-4f26-89af-cd304ca47a67" containerName="revoke-keys" Nov 02 08:28:48.850805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.850778 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:48.923166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.914808 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7"] Nov 02 08:28:48.961502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.960509 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:28:48.961502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.960638 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:28:48.961502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.960730 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:28:48.961502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.960793 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:28:48.976554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.969518 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:48.976554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.969702 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:48.976554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:48.969812 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khf6\" (UniqueName: \"kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.071185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.071032 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.071624 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.071595 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.071846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.071826 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khf6\" (UniqueName: \"kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.073528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.073494 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.076598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.076570 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.158221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.154679 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:28:49.158221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.154739 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:28:49.162196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.159294 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khf6\" (UniqueName: \"kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6\") pod \"maas-api-revoke-sub-6eb3548f-kzpz7\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.164505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.164479 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:49.546007 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.545953 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:49.546809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.546772 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:49.737197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:49.720207 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7"] Nov 02 08:28:49.737197 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:28:49.721798 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13335172_1491_4c6b_b75c_58fbc39bf141.slice/crio-7b1b5bf1c60f415e310d4919c31a1011e62ba37182ba4c49d64744a04e7d13ca WatchSource:0}: Error finding container 7b1b5bf1c60f415e310d4919c31a1011e62ba37182ba4c49d64744a04e7d13ca: Status 404 returned error can't find the container with id 7b1b5bf1c60f415e310d4919c31a1011e62ba37182ba4c49d64744a04e7d13ca Nov 02 08:28:50.147506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.147467 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:28:50.148006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.147530 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:28:50.545815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.545587 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:50.545815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.545655 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:50.573599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.573563 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" event={"ID":"13335172-1491-4c6b-b75c-58fbc39bf141","Type":"ContainerStarted","Data":"d5151fdefb22ec6e3b58be20cf86fd06dceb80d63647db57bc0d0c84a0ad6e36"} Nov 02 08:28:50.573780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.573657 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" event={"ID":"13335172-1491-4c6b-b75c-58fbc39bf141","Type":"ContainerStarted","Data":"7b1b5bf1c60f415e310d4919c31a1011e62ba37182ba4c49d64744a04e7d13ca"} Nov 02 08:28:50.701165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:50.694533 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" podStartSLOduration=2.6944950370000003 podStartE2EDuration="2.694495037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:50.691404788 +0000 UTC m=+1377.782025207" watchObservedRunningTime="2026-11-02 08:28:50.694495037 +0000 UTC m=+1377.785115457" Nov 02 08:28:51.147758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:51.147555 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:28:51.147758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:51.147607 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:28:51.545748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:51.545699 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:51.545972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:51.545757 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:51.610178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:51.608104 5055 generic.go:356] "Generic (PLEG): container finished" podID="13335172-1491-4c6b-b75c-58fbc39bf141" containerID="d5151fdefb22ec6e3b58be20cf86fd06dceb80d63647db57bc0d0c84a0ad6e36" exitCode=0 Nov 02 08:28:51.610178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:51.608418 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" event={"ID":"13335172-1491-4c6b-b75c-58fbc39bf141","Type":"ContainerDied","Data":"d5151fdefb22ec6e3b58be20cf86fd06dceb80d63647db57bc0d0c84a0ad6e36"} Nov 02 08:28:52.147052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:52.147018 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:28:52.147288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:52.147075 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:28:52.546900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:52.546732 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 08:28:52.546900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:52.546793 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 08:28:52.924084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:52.924046 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:53.025725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.025668 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca\") pod \"13335172-1491-4c6b-b75c-58fbc39bf141\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " Nov 02 08:28:53.026637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.026600 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6\" (UniqueName: \"kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6\") pod \"13335172-1491-4c6b-b75c-58fbc39bf141\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " Nov 02 08:28:53.027253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.027219 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca\") pod \"13335172-1491-4c6b-b75c-58fbc39bf141\" (UID: \"13335172-1491-4c6b-b75c-58fbc39bf141\") " Nov 02 08:28:53.027918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.027884 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca" pod "13335172-1491-4c6b-b75c-58fbc39bf141" (UID: "13335172-1491-4c6b-b75c-58fbc39bf141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:53.029251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.029231 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca" pod "13335172-1491-4c6b-b75c-58fbc39bf141" (UID: "13335172-1491-4c6b-b75c-58fbc39bf141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:53.035842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.035799 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6" pod "13335172-1491-4c6b-b75c-58fbc39bf141" (UID: "13335172-1491-4c6b-b75c-58fbc39bf141"). InnerVolumeSpecName "kube-api-access-5khf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:53.143899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.143807 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:53.144128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.144110 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5khf6\" (UniqueName: \"kubernetes.io/projected/13335172-1491-4c6b-b75c-58fbc39bf141-kube-api-access-5khf6\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:53.144316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.144301 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13335172-1491-4c6b-b75c-58fbc39bf141-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:53.151095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.151058 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:28:53.151311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.151112 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 08:28:53.153060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.153038 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b"] Nov 02 08:28:53.153784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.153761 5055 cpu_manager.go:414] "containerMap: removing container" podUID="13335172-1491-4c6b-b75c-58fbc39bf141" containerName="revoke-keys" Nov 02 08:28:53.153872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.153784 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13335172-1491-4c6b-b75c-58fbc39bf141" containerName="revoke-keys" Nov 02 08:28:53.153984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.153970 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="13335172-1491-4c6b-b75c-58fbc39bf141" containerName="revoke-keys" Nov 02 08:28:53.157284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.157262 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.169404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.169367 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b"] Nov 02 08:28:53.374171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.374095 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.374344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.374284 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.374344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.374324 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86qv\" (UniqueName: \"kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.476034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.475954 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.476289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.476095 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86qv\" (UniqueName: \"kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.476422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.476393 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.476962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.476923 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.478998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.478972 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.531714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.531680 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86qv\" (UniqueName: \"kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv\") pod \"maas-api-revoke-sub-94cbb200-xnf6b\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.658189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.657582 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:28:53.658189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.657610 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" event={"ID":"13335172-1491-4c6b-b75c-58fbc39bf141","Type":"ContainerDied","Data":"7b1b5bf1c60f415e310d4919c31a1011e62ba37182ba4c49d64744a04e7d13ca"} Nov 02 08:28:53.658189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.657686 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b1b5bf1c60f415e310d4919c31a1011e62ba37182ba4c49d64744a04e7d13ca" Nov 02 08:28:53.745058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.744955 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:53.746287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.746215 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:53.747708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.747682 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:28:53.802571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.802519 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:53.879166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:53.878304 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:28:54.152311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:54.152229 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:28:54.197189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:54.196180 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b"] Nov 02 08:28:54.666472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:54.666437 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" event={"ID":"dc9de232-a7ce-43a0-a1a6-add363636639","Type":"ContainerStarted","Data":"dec4cd53a5ca00a70048330f03a5814cd678d90c6ec4e5a082b058be8ebd7e26"} Nov 02 08:28:54.667691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:54.667673 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:28:54.920423 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:28:54.920331 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:28:55.674935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:55.674849 5055 generic.go:356] "Generic (PLEG): container finished" podID="dc9de232-a7ce-43a0-a1a6-add363636639" containerID="559ab60be65f98d50b78323dc681954b79d0b2b0e8352fc748fc35ea575a19dd" exitCode=0 Nov 02 08:28:55.675374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:55.674936 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" event={"ID":"dc9de232-a7ce-43a0-a1a6-add363636639","Type":"ContainerDied","Data":"559ab60be65f98d50b78323dc681954b79d0b2b0e8352fc748fc35ea575a19dd"} Nov 02 08:28:56.860967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.860941 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:56.927746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.927712 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv\" (UniqueName: \"kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv\") pod \"dc9de232-a7ce-43a0-a1a6-add363636639\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " Nov 02 08:28:56.927942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.927769 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca\") pod \"dc9de232-a7ce-43a0-a1a6-add363636639\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " Nov 02 08:28:56.927942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.927810 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca\") pod \"dc9de232-a7ce-43a0-a1a6-add363636639\" (UID: \"dc9de232-a7ce-43a0-a1a6-add363636639\") " Nov 02 08:28:56.928374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.928346 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca" pod "dc9de232-a7ce-43a0-a1a6-add363636639" (UID: "dc9de232-a7ce-43a0-a1a6-add363636639"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:56.930190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.930160 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca" pod "dc9de232-a7ce-43a0-a1a6-add363636639" (UID: "dc9de232-a7ce-43a0-a1a6-add363636639"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:56.930359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:56.930343 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv" pod "dc9de232-a7ce-43a0-a1a6-add363636639" (UID: "dc9de232-a7ce-43a0-a1a6-add363636639"). InnerVolumeSpecName "kube-api-access-f86qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:57.029922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:57.029885 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f86qv\" (UniqueName: \"kubernetes.io/projected/dc9de232-a7ce-43a0-a1a6-add363636639-kube-api-access-f86qv\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:57.029922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:57.029915 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:57.029922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:57.029927 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9de232-a7ce-43a0-a1a6-add363636639-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:28:57.702344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:57.702313 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" Nov 02 08:28:57.702533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:57.702353 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b" event={"ID":"dc9de232-a7ce-43a0-a1a6-add363636639","Type":"ContainerDied","Data":"dec4cd53a5ca00a70048330f03a5814cd678d90c6ec4e5a082b058be8ebd7e26"} Nov 02 08:28:57.702533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:57.702437 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec4cd53a5ca00a70048330f03a5814cd678d90c6ec4e5a082b058be8ebd7e26" Nov 02 08:28:59.317179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.317133 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf"] Nov 02 08:28:59.317740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.317725 5055 cpu_manager.go:414] "containerMap: removing container" podUID="dc9de232-a7ce-43a0-a1a6-add363636639" containerName="revoke-keys" Nov 02 08:28:59.317802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.317740 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9de232-a7ce-43a0-a1a6-add363636639" containerName="revoke-keys" Nov 02 08:28:59.317869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.317859 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9de232-a7ce-43a0-a1a6-add363636639" containerName="revoke-keys" Nov 02 08:28:59.320688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.320668 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363398 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363456 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363586 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363650 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363691 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363807 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363892 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.363940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.363918 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.364047 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.364028 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj96w\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.465573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465538 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.465724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465596 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.465724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465634 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.465724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465679 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.465886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465750 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.465886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465840 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.465869 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466005 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj96w\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466105 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466185 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466200 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466484 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466500 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.466798 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.466779 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.468285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.468262 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.468426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.468411 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.509712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.509671 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf"] Nov 02 08:28:59.524201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.521134 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.524201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.521366 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj96w\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:28:59.633732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:28:59.633655 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:29:00.062166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.062115 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf"] Nov 02 08:29:00.063551 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:29:00.063522 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801546db_f697_46a3_a7df_cc99ae52131b.slice/crio-12f73d516096a1c6782d8b5b34ac6f9d6fa8ee61cf3cfaafc14b130f2b4eaa7a WatchSource:0}: Error finding container 12f73d516096a1c6782d8b5b34ac6f9d6fa8ee61cf3cfaafc14b130f2b4eaa7a: Status 404 returned error can't find the container with id 12f73d516096a1c6782d8b5b34ac6f9d6fa8ee61cf3cfaafc14b130f2b4eaa7a Nov 02 08:29:00.066253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.066221 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:29:00.066343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.066298 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:29:00.066396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.066343 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:29:00.730028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.729991 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" event={"ID":"801546db-f697-46a3-a7df-cc99ae52131b","Type":"ContainerStarted","Data":"5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5"} Nov 02 08:29:00.730470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.730062 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" event={"ID":"801546db-f697-46a3-a7df-cc99ae52131b","Type":"ContainerStarted","Data":"12f73d516096a1c6782d8b5b34ac6f9d6fa8ee61cf3cfaafc14b130f2b4eaa7a"} Nov 02 08:29:00.905641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:00.905580 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podStartSLOduration=1.905566112 podStartE2EDuration="1.905566112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:00.904537157 +0000 UTC m=+1387.995157579" watchObservedRunningTime="2026-11-02 08:29:00.905566112 +0000 UTC m=+1387.996186531" Nov 02 08:29:01.634207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:01.634175 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:29:01.636255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:01.636217 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 08:29:01.636404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:01.636292 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 08:29:02.634086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:02.634028 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 08:29:02.634749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:02.634704 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 08:29:03.634326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:03.634285 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 08:29:03.634755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:03.634349 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 08:29:04.634211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:04.634177 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 08:29:04.634374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:04.634223 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 08:29:05.634654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:05.634624 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 08:29:05.635125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:05.635094 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 08:29:06.634183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:06.634108 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 08:29:06.634391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:06.634217 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 08:29:07.639376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:07.639343 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:29:07.639811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:07.639797 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:29:07.640617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:07.640595 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:29:11.051512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:11.051472 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd"] Nov 02 08:29:11.097558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:11.097520 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ccb32d-2pnwd"] Nov 02 08:29:13.095754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:13.095717 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd005872-8df4-49e0-9710-ca51e6908619" path="/var/lib/kubelet/pods/cd005872-8df4-49e0-9710-ca51e6908619/volumes" Nov 02 08:29:14.818960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.818917 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb"] Nov 02 08:29:14.823750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.823723 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:14.872579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.872543 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:29:14.919315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.918662 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:29:14.919315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.918740 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:29:14.919315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.918862 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:29:14.925999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.925967 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb"] Nov 02 08:29:14.931245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.931212 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:14.931590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.931567 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:14.931646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:14.931618 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsb7\" (UniqueName: \"kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.032647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.032609 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsb7\" (UniqueName: \"kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.032840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.032671 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.032941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.032922 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.033622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.033595 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.035136 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.035107 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.087493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.087385 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsb7\" (UniqueName: \"kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7\") pod \"maas-api-revoke-sub-838627ed-t64nb\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.140469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.140434 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:15.526225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.526185 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb"] Nov 02 08:29:15.526947 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:29:15.526917 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae80f6a0_7bcd_489f_a6e8_7eb0146757cd.slice/crio-a21060247c240c8f8221382c5976fb5211384c6b468c8bde54c150637c3fffc3 WatchSource:0}: Error finding container a21060247c240c8f8221382c5976fb5211384c6b468c8bde54c150637c3fffc3: Status 404 returned error can't find the container with id a21060247c240c8f8221382c5976fb5211384c6b468c8bde54c150637c3fffc3 Nov 02 08:29:15.860444 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:15.860358 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" event={"ID":"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd","Type":"ContainerStarted","Data":"a21060247c240c8f8221382c5976fb5211384c6b468c8bde54c150637c3fffc3"} Nov 02 08:29:16.869775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:16.869685 5055 generic.go:356] "Generic (PLEG): container finished" podID="ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" containerID="c426cc900aab91fa99982a2d5dd6168faa189dc012dafd92a59ccada6ace090a" exitCode=0 Nov 02 08:29:16.870285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:16.869830 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" event={"ID":"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd","Type":"ContainerDied","Data":"c426cc900aab91fa99982a2d5dd6168faa189dc012dafd92a59ccada6ace090a"} Nov 02 08:29:18.069741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.069700 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:18.178404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.178225 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca\") pod \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " Nov 02 08:29:18.178404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.178272 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7\" (UniqueName: \"kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7\") pod \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " Nov 02 08:29:18.178728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.178458 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca\") pod \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\" (UID: \"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd\") " Nov 02 08:29:18.178728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.178648 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca" pod "ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" (UID: "ae80f6a0-7bcd-489f-a6e8-7eb0146757cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:29:18.179364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.179332 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:18.193804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.193759 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7" pod "ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" (UID: "ae80f6a0-7bcd-489f-a6e8-7eb0146757cd"). InnerVolumeSpecName "kube-api-access-9xsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:29:18.193804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.193779 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca" pod "ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" (UID: "ae80f6a0-7bcd-489f-a6e8-7eb0146757cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:29:18.280890 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.280850 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xsb7\" (UniqueName: \"kubernetes.io/projected/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-kube-api-access-9xsb7\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:18.280890 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.280886 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:18.890994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.890958 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" event={"ID":"ae80f6a0-7bcd-489f-a6e8-7eb0146757cd","Type":"ContainerDied","Data":"a21060247c240c8f8221382c5976fb5211384c6b468c8bde54c150637c3fffc3"} Nov 02 08:29:18.891244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.891016 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21060247c240c8f8221382c5976fb5211384c6b468c8bde54c150637c3fffc3" Nov 02 08:29:18.891244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:18.890966 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb" Nov 02 08:29:19.060825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:19.060795 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:29:21.410171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:21.409254 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:21.467165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:21.466491 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:21.626177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:21.624628 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:21.770165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:21.769223 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:21.792255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:21.792210 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:22.296720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.296668 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 08:29:22.383808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.383239 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:22.385030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.384298 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" podUID="5d13bf2a-d174-4b83-b363-ed42f963cff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 02 08:29:22.416177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.414153 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:22.416177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.414555 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:29:22.447779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.447747 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:29:22.704225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:22.673240 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:24.926720 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:29:24.926674 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:29:25.517280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.517248 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9"] Nov 02 08:29:25.517867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.517851 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" containerName="revoke-keys" Nov 02 08:29:25.517949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.517868 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" containerName="revoke-keys" Nov 02 08:29:25.518014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.518004 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" containerName="revoke-keys" Nov 02 08:29:25.522973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.522945 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.575469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.575435 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9"] Nov 02 08:29:25.615794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.615757 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:29:25.616075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.615769 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:29:25.616172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.615823 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:29:25.617733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.617711 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:29:25.667701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.667666 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6rx\" (UniqueName: \"kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.667878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.667812 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.667878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.667846 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.768891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.768794 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.768891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.768851 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.769136 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.768991 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6rx\" (UniqueName: \"kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.769574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.769549 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.771333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.771313 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.823647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.823611 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6rx\" (UniqueName: \"kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx\") pod \"maas-api-revoke-sub-224327da-msqz9\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:25.837510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:25.837478 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:26.222528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:26.222491 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9"] Nov 02 08:29:26.222954 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:29:26.222716 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595f472a_4b68_408d_b382_71d104b1f0c5.slice/crio-b6115eb0465d178a445f814cb9659950c2ce248f0253bf285474d0bd5b48285a WatchSource:0}: Error finding container b6115eb0465d178a445f814cb9659950c2ce248f0253bf285474d0bd5b48285a: Status 404 returned error can't find the container with id b6115eb0465d178a445f814cb9659950c2ce248f0253bf285474d0bd5b48285a Nov 02 08:29:27.223250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:27.223215 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" event={"ID":"595f472a-4b68-408d-b382-71d104b1f0c5","Type":"ContainerStarted","Data":"b904997135230f5454ab8b762e2ba6df011497f9101ff1168c7be48998f74b52"} Nov 02 08:29:27.223659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:27.223272 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" event={"ID":"595f472a-4b68-408d-b382-71d104b1f0c5","Type":"ContainerStarted","Data":"b6115eb0465d178a445f814cb9659950c2ce248f0253bf285474d0bd5b48285a"} Nov 02 08:29:27.333427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:27.333367 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" podStartSLOduration=2.333352567 podStartE2EDuration="2.333352567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:27.329410443 +0000 UTC m=+1414.420030863" watchObservedRunningTime="2026-11-02 08:29:27.333352567 +0000 UTC m=+1414.423973014" Nov 02 08:29:28.235967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:28.235924 5055 generic.go:356] "Generic (PLEG): container finished" podID="595f472a-4b68-408d-b382-71d104b1f0c5" containerID="b904997135230f5454ab8b762e2ba6df011497f9101ff1168c7be48998f74b52" exitCode=0 Nov 02 08:29:28.236467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:28.236002 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" event={"ID":"595f472a-4b68-408d-b382-71d104b1f0c5","Type":"ContainerDied","Data":"b904997135230f5454ab8b762e2ba6df011497f9101ff1168c7be48998f74b52"} Nov 02 08:29:29.448922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.448889 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:29.628730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.628633 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca\") pod \"595f472a-4b68-408d-b382-71d104b1f0c5\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " Nov 02 08:29:29.628897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.628800 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca\") pod \"595f472a-4b68-408d-b382-71d104b1f0c5\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " Nov 02 08:29:29.628962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.628907 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx\" (UniqueName: \"kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx\") pod \"595f472a-4b68-408d-b382-71d104b1f0c5\" (UID: \"595f472a-4b68-408d-b382-71d104b1f0c5\") " Nov 02 08:29:29.629092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.629063 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca" pod "595f472a-4b68-408d-b382-71d104b1f0c5" (UID: "595f472a-4b68-408d-b382-71d104b1f0c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:29:29.629557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.629532 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:29.631084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.631054 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx" pod "595f472a-4b68-408d-b382-71d104b1f0c5" (UID: "595f472a-4b68-408d-b382-71d104b1f0c5"). InnerVolumeSpecName "kube-api-access-xs6rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:29:29.631246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.631118 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca" pod "595f472a-4b68-408d-b382-71d104b1f0c5" (UID: "595f472a-4b68-408d-b382-71d104b1f0c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:29:29.730545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.730503 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595f472a-4b68-408d-b382-71d104b1f0c5-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:29.730545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:29.730536 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6rx\" (UniqueName: \"kubernetes.io/projected/595f472a-4b68-408d-b382-71d104b1f0c5-kube-api-access-xs6rx\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:30.251613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:30.251583 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" Nov 02 08:29:30.251613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:30.251602 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9" event={"ID":"595f472a-4b68-408d-b382-71d104b1f0c5","Type":"ContainerDied","Data":"b6115eb0465d178a445f814cb9659950c2ce248f0253bf285474d0bd5b48285a"} Nov 02 08:29:30.251861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:30.251660 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6115eb0465d178a445f814cb9659950c2ce248f0253bf285474d0bd5b48285a" Nov 02 08:29:31.708843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.708802 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2"] Nov 02 08:29:31.709747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.709724 5055 cpu_manager.go:414] "containerMap: removing container" podUID="595f472a-4b68-408d-b382-71d104b1f0c5" containerName="revoke-keys" Nov 02 08:29:31.709838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.709749 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595f472a-4b68-408d-b382-71d104b1f0c5" containerName="revoke-keys" Nov 02 08:29:31.709952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.709939 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="595f472a-4b68-408d-b382-71d104b1f0c5" containerName="revoke-keys" Nov 02 08:29:31.714568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.714542 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.756235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.756196 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.756423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.756397 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmwj\" (UniqueName: \"kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.756517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.756504 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.764353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.764326 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2"] Nov 02 08:29:31.805197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.805136 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:29:31.805197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.805185 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:29:31.805420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.805171 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:29:31.809176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.809152 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:29:31.857648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.857604 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.857828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.857772 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmwj\" (UniqueName: \"kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.857933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.857918 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.858423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.858396 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.860400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.860382 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:31.911732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:31.911703 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmwj\" (UniqueName: \"kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj\") pod \"maas-api-revoke-sub-cd78a964-g6lf2\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:32.031238 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:32.031153 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:32.407068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:32.407034 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2"] Nov 02 08:29:32.407522 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:29:32.407497 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a2fa93_b7dd_4d19_a806_191475ea3e4c.slice/crio-5d72540f479dd03edba337a16cebedebbb4721eec8265a3c75c7a174bb17f160 WatchSource:0}: Error finding container 5d72540f479dd03edba337a16cebedebbb4721eec8265a3c75c7a174bb17f160: Status 404 returned error can't find the container with id 5d72540f479dd03edba337a16cebedebbb4721eec8265a3c75c7a174bb17f160 Nov 02 08:29:33.280779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:33.280739 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" event={"ID":"b1a2fa93-b7dd-4d19-a806-191475ea3e4c","Type":"ContainerStarted","Data":"8495f278397afd3b5da554a3e8ddac6b0a8616b5b7c093d4e2be7a68229a3ae7"} Nov 02 08:29:33.281248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:33.280807 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" event={"ID":"b1a2fa93-b7dd-4d19-a806-191475ea3e4c","Type":"ContainerStarted","Data":"5d72540f479dd03edba337a16cebedebbb4721eec8265a3c75c7a174bb17f160"} Nov 02 08:29:33.436987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:33.436925 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" podStartSLOduration=2.436908452 podStartE2EDuration="2.436908452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:33.38700772 +0000 UTC m=+1420.477628151" watchObservedRunningTime="2026-11-02 08:29:33.436908452 +0000 UTC m=+1420.527528870" Nov 02 08:29:34.291335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:34.291294 5055 generic.go:356] "Generic (PLEG): container finished" podID="b1a2fa93-b7dd-4d19-a806-191475ea3e4c" containerID="8495f278397afd3b5da554a3e8ddac6b0a8616b5b7c093d4e2be7a68229a3ae7" exitCode=0 Nov 02 08:29:34.291746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:34.291333 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" event={"ID":"b1a2fa93-b7dd-4d19-a806-191475ea3e4c","Type":"ContainerDied","Data":"8495f278397afd3b5da554a3e8ddac6b0a8616b5b7c093d4e2be7a68229a3ae7"} Nov 02 08:29:35.491066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.491037 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:35.609096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.609021 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca\") pod \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " Nov 02 08:29:35.609302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.609282 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca\") pod \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " Nov 02 08:29:35.609471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.609441 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca" pod "b1a2fa93-b7dd-4d19-a806-191475ea3e4c" (UID: "b1a2fa93-b7dd-4d19-a806-191475ea3e4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:29:35.609540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.609452 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj\" (UniqueName: \"kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj\") pod \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\" (UID: \"b1a2fa93-b7dd-4d19-a806-191475ea3e4c\") " Nov 02 08:29:35.610182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.610156 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:35.612309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.612266 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca" pod "b1a2fa93-b7dd-4d19-a806-191475ea3e4c" (UID: "b1a2fa93-b7dd-4d19-a806-191475ea3e4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:29:35.612309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.612290 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj" pod "b1a2fa93-b7dd-4d19-a806-191475ea3e4c" (UID: "b1a2fa93-b7dd-4d19-a806-191475ea3e4c"). InnerVolumeSpecName "kube-api-access-7mmwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:29:35.722060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.722009 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:35.722060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:35.722057 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmwj\" (UniqueName: \"kubernetes.io/projected/b1a2fa93-b7dd-4d19-a806-191475ea3e4c-kube-api-access-7mmwj\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:36.314327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:36.314289 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" event={"ID":"b1a2fa93-b7dd-4d19-a806-191475ea3e4c","Type":"ContainerDied","Data":"5d72540f479dd03edba337a16cebedebbb4721eec8265a3c75c7a174bb17f160"} Nov 02 08:29:36.314510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:36.314352 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d72540f479dd03edba337a16cebedebbb4721eec8265a3c75c7a174bb17f160" Nov 02 08:29:36.314510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:36.314422 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2" Nov 02 08:29:37.094847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.094815 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll"] Nov 02 08:29:37.096298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.096267 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b1a2fa93-b7dd-4d19-a806-191475ea3e4c" containerName="revoke-keys" Nov 02 08:29:37.096489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.096474 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a2fa93-b7dd-4d19-a806-191475ea3e4c" containerName="revoke-keys" Nov 02 08:29:37.097027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.097008 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a2fa93-b7dd-4d19-a806-191475ea3e4c" containerName="revoke-keys" Nov 02 08:29:37.102661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.102635 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142130 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142384 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142224 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142384 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142330 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142524 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142408 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142637 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5c6l\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142824 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142707 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.142970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.142945 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.143057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.143041 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.143201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.143183 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244425 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244558 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5c6l\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244637 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244668 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244760 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244823 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244874 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.244932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.244915 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.245331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.245306 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.245939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.245908 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.246789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.246528 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.246789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.246629 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.246954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.246732 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.247596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.247574 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.247678 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.247607 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.299959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.299898 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.301636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.301603 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll"] Nov 02 08:29:37.302756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.302728 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5c6l\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.418347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.418255 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:37.852573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.852387 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll"] Nov 02 08:29:37.853412 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:29:37.853373 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc800c0aa_716d_40aa_8709_70274d48d9d8.slice/crio-a57d65646d4aad83c490c94e7182382469e0256bf6c22c3def601cb6d34032a4 WatchSource:0}: Error finding container a57d65646d4aad83c490c94e7182382469e0256bf6c22c3def601cb6d34032a4: Status 404 returned error can't find the container with id a57d65646d4aad83c490c94e7182382469e0256bf6c22c3def601cb6d34032a4 Nov 02 08:29:37.856103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.856061 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:29:37.856248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.856163 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:29:37.856248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:37.856200 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:29:38.338014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:38.337964 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" event={"ID":"c800c0aa-716d-40aa-8709-70274d48d9d8","Type":"ContainerStarted","Data":"316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e"} Nov 02 08:29:38.338465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:38.338042 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" event={"ID":"c800c0aa-716d-40aa-8709-70274d48d9d8","Type":"ContainerStarted","Data":"a57d65646d4aad83c490c94e7182382469e0256bf6c22c3def601cb6d34032a4"} Nov 02 08:29:38.418743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:38.418705 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:38.420335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:38.420305 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:38.420471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:38.420358 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:38.518341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:38.518277 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podStartSLOduration=1.518258452 podStartE2EDuration="1.518258452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:38.50370245 +0000 UTC m=+1425.594322870" watchObservedRunningTime="2026-11-02 08:29:38.518258452 +0000 UTC m=+1425.608878877" Nov 02 08:29:39.117048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:39.116312 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" podUID="34d28ca7a91586006c95d24bde05028b" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:29:39.418961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:39.418871 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:39.418961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:39.418927 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:40.418712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:40.418669 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:40.418890 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:40.418733 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:41.419382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:41.419341 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:41.419803 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:41.419407 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:42.418738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.418700 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:42.418923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.418766 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:42.469398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.469297 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z"] Nov 02 08:29:42.473598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.473574 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.530120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.530082 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z"] Nov 02 08:29:42.568651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.568608 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:29:42.568651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.568627 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:29:42.568856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.568624 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:29:42.568856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.568670 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:29:42.639757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.639722 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.639982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.639959 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.640056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.640021 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthpf\" (UniqueName: \"kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.741251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.741212 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.741472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.741284 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthpf\" (UniqueName: \"kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.741472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.741351 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.742069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.742035 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.743787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.743760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.801164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.797950 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthpf\" (UniqueName: \"kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf\") pod \"maas-api-revoke-sub-d0e4de34-zxx7z\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:42.820573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:42.820527 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:43.254499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:43.253060 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z"] Nov 02 08:29:43.377080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:43.377048 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" event={"ID":"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3","Type":"ContainerStarted","Data":"3c42966092cb98acb82b86c55c4d0c38d26cb6cfb9885709a1cbbecaca5d7574"} Nov 02 08:29:43.419161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:43.419115 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:43.419311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:43.419185 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:44.392925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:44.392888 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" event={"ID":"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3","Type":"ContainerStarted","Data":"29d2d9abdab8b69e4b83f8b445c4f2ac7c45ba2dcd7fb9ed51f788f59558cc89"} Nov 02 08:29:44.418760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:44.418717 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:29:44.418957 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:44.418787 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 08:29:44.506591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:44.506518 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" podStartSLOduration=2.506498027 podStartE2EDuration="2.506498027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:44.500089585 +0000 UTC m=+1431.590710005" watchObservedRunningTime="2026-11-02 08:29:44.506498027 +0000 UTC m=+1431.597118447" Nov 02 08:29:45.403076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.403036 5055 generic.go:356] "Generic (PLEG): container finished" podID="bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" containerID="29d2d9abdab8b69e4b83f8b445c4f2ac7c45ba2dcd7fb9ed51f788f59558cc89" exitCode=0 Nov 02 08:29:45.403649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.403175 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" event={"ID":"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3","Type":"ContainerDied","Data":"29d2d9abdab8b69e4b83f8b445c4f2ac7c45ba2dcd7fb9ed51f788f59558cc89"} Nov 02 08:29:45.456337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.456297 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:45.456737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.456713 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:45.457549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.457518 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:29:45.621691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.621654 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp"] Nov 02 08:29:45.626532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.626505 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.678214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.677178 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cba6f9af\"/\"default-dockercfg-4x8ql\"" Nov 02 08:29:45.678214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.677281 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cba6f9af\"/\"e2e-lifecycle-model-cba6f9af-kserve-self-signed-certs\"" Nov 02 08:29:45.678214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.677389 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cba6f9af\"/\"openshift-service-ca.crt\"" Nov 02 08:29:45.678214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.677469 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cba6f9af\"/\"kube-root-ca.crt\"" Nov 02 08:29:45.687500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.687463 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp"] Nov 02 08:29:45.823982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.811712 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.823982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.811827 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr84p\" (UniqueName: \"kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.823982 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.811893 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.827928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.827889 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.828095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.828005 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.828235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.828205 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.929949 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr84p\" (UniqueName: \"kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930013 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930078 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930133 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930282 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930448 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930398 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930448 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930418 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930489 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.930627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.930612 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.932687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.932655 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.932892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.932870 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:45.983714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:45.983678 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr84p\" (UniqueName: \"kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p\") pod \"e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:46.239429 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.239391 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:46.567417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.567388 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:46.633518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.633491 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp"] Nov 02 08:29:46.636236 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:29:46.636198 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6af0b3b_1a28_40f8_ac81_7fe5a440f968.slice/crio-020ae4ef3dd9da17a050fdf5b655493c3c543ed74d61c5bb0b9ed6411f266658 WatchSource:0}: Error finding container 020ae4ef3dd9da17a050fdf5b655493c3c543ed74d61c5bb0b9ed6411f266658: Status 404 returned error can't find the container with id 020ae4ef3dd9da17a050fdf5b655493c3c543ed74d61c5bb0b9ed6411f266658 Nov 02 08:29:46.638556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.638531 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca\") pod \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " Nov 02 08:29:46.638803 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.638733 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca\") pod \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " Nov 02 08:29:46.638849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.638812 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf\" (UniqueName: \"kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf\") pod \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\" (UID: \"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3\") " Nov 02 08:29:46.638921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.638900 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca" pod "bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" (UID: "bcf9096f-fa91-4819-a24a-a7eb15ec5ba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:29:46.639411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.639385 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:46.640851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.640819 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca" pod "bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" (UID: "bcf9096f-fa91-4819-a24a-a7eb15ec5ba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:29:46.640953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.640898 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf" pod "bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" (UID: "bcf9096f-fa91-4819-a24a-a7eb15ec5ba3"). InnerVolumeSpecName "kube-api-access-qthpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:29:46.741011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.740975 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:46.741011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:46.741003 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qthpf\" (UniqueName: \"kubernetes.io/projected/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3-kube-api-access-qthpf\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:29:47.423516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:47.423485 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" Nov 02 08:29:47.423737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:47.423484 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z" event={"ID":"bcf9096f-fa91-4819-a24a-a7eb15ec5ba3","Type":"ContainerDied","Data":"3c42966092cb98acb82b86c55c4d0c38d26cb6cfb9885709a1cbbecaca5d7574"} Nov 02 08:29:47.423737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:47.423656 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c42966092cb98acb82b86c55c4d0c38d26cb6cfb9885709a1cbbecaca5d7574" Nov 02 08:29:47.425634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:47.425604 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" event={"ID":"c6af0b3b-1a28-40f8-ac81-7fe5a440f968","Type":"ContainerStarted","Data":"b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c"} Nov 02 08:29:47.425970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:47.425941 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" event={"ID":"c6af0b3b-1a28-40f8-ac81-7fe5a440f968","Type":"ContainerStarted","Data":"020ae4ef3dd9da17a050fdf5b655493c3c543ed74d61c5bb0b9ed6411f266658"} Nov 02 08:29:47.534694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:47.534640 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" podStartSLOduration=2.534625477 podStartE2EDuration="2.534625477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:47.534252801 +0000 UTC m=+1434.624873234" watchObservedRunningTime="2026-11-02 08:29:47.534625477 +0000 UTC m=+1434.625245896" Nov 02 08:29:48.239540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:48.239506 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:53.528823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:53.528786 5055 scope.go:122] "RemoveContainer" containerID="3c2d7781ff3c81beab9b80c1c66650aa575eafea21e13b633e770f14c9ba01c0" Nov 02 08:29:55.112164 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:29:55.101841 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:29:56.239879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:56.239685 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:56.256629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:56.256597 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:29:56.655710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:29:56.655618 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:30:00.161071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.161029 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2"] Nov 02 08:30:00.162080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.162055 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" containerName="revoke-keys" Nov 02 08:30:00.162173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.162080 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" containerName="revoke-keys" Nov 02 08:30:00.162328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.162313 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" containerName="revoke-keys" Nov 02 08:30:00.166673 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.166647 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl"] Nov 02 08:30:00.166812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.166779 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.171824 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.171796 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8"] Nov 02 08:30:00.171966 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.171919 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.177075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.177049 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.217242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.217213 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg"] Nov 02 08:30:00.225998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.225966 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd"] Nov 02 08:30:00.226182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.226098 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.231575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.231546 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr"] Nov 02 08:30:00.231775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.231643 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.236820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.236795 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz"] Nov 02 08:30:00.236949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.236905 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.242384 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.242129 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr"] Nov 02 08:30:00.243194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.242506 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.249305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.248003 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:30:00.249305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.248204 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:30:00.249305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.248306 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:30:00.251634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.251611 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.263240 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.263212 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8td5j\" (UniqueName: \"kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.263378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.263346 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.263468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.263451 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrlz\" (UniqueName: \"kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz\") pod \"maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.263537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.263520 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.263632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.263614 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.263710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.263671 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9tk\" (UniqueName: \"kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.267904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.267877 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2"] Nov 02 08:30:00.318918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.315812 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl"] Nov 02 08:30:00.336875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.336848 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8"] Nov 02 08:30:00.346493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.346423 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd"] Nov 02 08:30:00.354308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.354243 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr"] Nov 02 08:30:00.360329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.360299 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz"] Nov 02 08:30:00.372220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.372185 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr"] Nov 02 08:30:00.374002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.373969 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwts\" (UniqueName: \"kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.374204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374034 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8td5j\" (UniqueName: \"kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.374204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374175 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.374353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374219 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.374353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374272 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls294\" (UniqueName: \"kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294\") pod \"maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.374353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374320 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67h78\" (UniqueName: \"kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78\") pod \"maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.374533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374381 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrlz\" (UniqueName: \"kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz\") pod \"maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.374533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374429 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.374533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374479 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.374533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374506 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.374772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374559 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8\") pod \"maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.374772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374591 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.374772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374615 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.374772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374647 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.374772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374697 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9tk\" (UniqueName: \"kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.374772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.374726 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6g4\" (UniqueName: \"kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4\") pod \"maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.375965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.375934 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.376935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.376904 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.377367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.377341 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.378388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.378365 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg"] Nov 02 08:30:00.432003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.431968 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9tk\" (UniqueName: \"kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.432652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.432631 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrlz\" (UniqueName: \"kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz\") pod \"maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.433042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.433025 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8td5j\" (UniqueName: \"kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.476779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.476731 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.476975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.476790 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.476975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.476847 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8\") pod \"maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.476975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.476879 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.476975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.476912 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.476975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.476963 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6g4\" (UniqueName: \"kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4\") pod \"maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.477270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477040 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwts\" (UniqueName: \"kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.477270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477193 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.477374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477273 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls294\" (UniqueName: \"kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294\") pod \"maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.477374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477327 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67h78\" (UniqueName: \"kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78\") pod \"maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.477702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477672 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.477808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477685 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.477989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.477968 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.478425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.478403 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.478477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.478449 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.488187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.486915 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:00.509954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.509918 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:00.521181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.520885 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:00.534322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.534293 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8\") pod \"maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.534974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.534924 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6g4\" (UniqueName: \"kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4\") pod \"maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.536391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.536344 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67h78\" (UniqueName: \"kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78\") pod \"maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.536391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.536359 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls294\" (UniqueName: \"kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294\") pod \"maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.538077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.538054 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwts\" (UniqueName: \"kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.545174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.544916 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:00.562426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.562394 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:00.602204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.600416 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:00.619714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.619534 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:00.655346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.652493 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:00.913657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:00.913628 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2"] Nov 02 08:30:01.173082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.173054 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8"] Nov 02 08:30:01.184230 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:01.184186 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997c95e9_13d4_4528_b755_d7b070a1e15f.slice/crio-2619b4ed6fdcced24a7d92800aa8e6a930915fc93ea5cfe9d57b643a40192fba WatchSource:0}: Error finding container 2619b4ed6fdcced24a7d92800aa8e6a930915fc93ea5cfe9d57b643a40192fba: Status 404 returned error can't find the container with id 2619b4ed6fdcced24a7d92800aa8e6a930915fc93ea5cfe9d57b643a40192fba Nov 02 08:30:01.241939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.241910 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl"] Nov 02 08:30:01.242097 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:01.242008 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce6934c_5a76_4ff6_a9fe_60adffaa6f4c.slice/crio-744b476f98822d10cccfb0eb9aff21191843cf55b9bc390ce1d0f83ba85df5ba WatchSource:0}: Error finding container 744b476f98822d10cccfb0eb9aff21191843cf55b9bc390ce1d0f83ba85df5ba: Status 404 returned error can't find the container with id 744b476f98822d10cccfb0eb9aff21191843cf55b9bc390ce1d0f83ba85df5ba Nov 02 08:30:01.334761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.334724 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg"] Nov 02 08:30:01.356071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.355791 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz"] Nov 02 08:30:01.360398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.360370 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr"] Nov 02 08:30:01.388202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.388168 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd"] Nov 02 08:30:01.404179 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:01.404110 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88030485_a961_4f5d_bab9_f592f564af2d.slice/crio-8147aa64975c588e78e49244d1bf5c0c4ebb66187834a2616d7d2f51c6bdad59 WatchSource:0}: Error finding container 8147aa64975c588e78e49244d1bf5c0c4ebb66187834a2616d7d2f51c6bdad59: Status 404 returned error can't find the container with id 8147aa64975c588e78e49244d1bf5c0c4ebb66187834a2616d7d2f51c6bdad59 Nov 02 08:30:01.405913 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:01.405878 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0eda2_617d_41f0_a62f_8ea2e2b5300c.slice/crio-c8a53a5b3b0f0c58bd7e2312a37f1f78098d1a476b428a226dabf6ad0117166e WatchSource:0}: Error finding container c8a53a5b3b0f0c58bd7e2312a37f1f78098d1a476b428a226dabf6ad0117166e: Status 404 returned error can't find the container with id c8a53a5b3b0f0c58bd7e2312a37f1f78098d1a476b428a226dabf6ad0117166e Nov 02 08:30:01.428267 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.428235 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr"] Nov 02 08:30:01.688051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.687956 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerStarted","Data":"edc2d4117268c909bd6f116d82360e48b683cb0ce46d77f39b6a69fdb97bdba2"} Nov 02 08:30:01.690412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.690323 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerStarted","Data":"b8f01ec8b51fa31d307d53d6cb1824484f1b28c16cb8b4d2407b8e76efb7dc7d"} Nov 02 08:30:01.690412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.690376 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerStarted","Data":"2619b4ed6fdcced24a7d92800aa8e6a930915fc93ea5cfe9d57b643a40192fba"} Nov 02 08:30:01.692454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.692427 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerStarted","Data":"744b476f98822d10cccfb0eb9aff21191843cf55b9bc390ce1d0f83ba85df5ba"} Nov 02 08:30:01.694097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.694033 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" event={"ID":"240de6f6-cb10-437e-a9dd-122ba69b21b3","Type":"ContainerStarted","Data":"ee0162b199673b8799c24d5f2929d920f147d3ecc5e3174121d30db1ea931cdc"} Nov 02 08:30:01.696058 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.696032 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerStarted","Data":"c7f3e36589e5eba33686630bbd6adb52ee349896cba35a658affa6d60613f8ee"} Nov 02 08:30:01.696262 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.696241 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerStarted","Data":"7eed16c127855a0da83d647a23680268ab63698f519594b69e80b115d0ecbc14"} Nov 02 08:30:01.698538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.698511 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerStarted","Data":"c8a53a5b3b0f0c58bd7e2312a37f1f78098d1a476b428a226dabf6ad0117166e"} Nov 02 08:30:01.700661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.700639 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerStarted","Data":"dbf89b97fb144be012ff031a842b0c669a544184b98545a41b74007d7c70678f"} Nov 02 08:30:01.703505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.703483 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerStarted","Data":"8147aa64975c588e78e49244d1bf5c0c4ebb66187834a2616d7d2f51c6bdad59"} Nov 02 08:30:01.800399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.800339 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" podStartSLOduration=1.8003208769999999 podStartE2EDuration="1.800320877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:01.797315307 +0000 UTC m=+1448.887935728" watchObservedRunningTime="2026-11-02 08:30:01.800320877 +0000 UTC m=+1448.890941298" Nov 02 08:30:01.914005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:01.913913 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" podStartSLOduration=1.9138866490000002 podStartE2EDuration="1.913886649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:01.907317425 +0000 UTC m=+1448.997937844" watchObservedRunningTime="2026-11-02 08:30:01.913886649 +0000 UTC m=+1449.004507069" Nov 02 08:30:02.743788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.743383 5055 generic.go:356] "Generic (PLEG): container finished" podID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerID="c7f3e36589e5eba33686630bbd6adb52ee349896cba35a658affa6d60613f8ee" exitCode=60 Nov 02 08:30:02.743788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.743451 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerDied","Data":"c7f3e36589e5eba33686630bbd6adb52ee349896cba35a658affa6d60613f8ee"} Nov 02 08:30:02.743788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.743795 5055 scope.go:122] "RemoveContainer" containerID="c7f3e36589e5eba33686630bbd6adb52ee349896cba35a658affa6d60613f8ee" Nov 02 08:30:02.747176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.746676 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerStarted","Data":"adbdc00102a115bbab795b3a1b4905ece8fe4a7f4961b973e9054a158b2b5ad8"} Nov 02 08:30:02.747176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.746927 5055 scope.go:122] "RemoveContainer" containerID="adbdc00102a115bbab795b3a1b4905ece8fe4a7f4961b973e9054a158b2b5ad8" Nov 02 08:30:02.748825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.748799 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerStarted","Data":"2a9443d3d04def9a9e5177a0690878603397a25c450bf27c3353f92f3a70658a"} Nov 02 08:30:02.749182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.749161 5055 scope.go:122] "RemoveContainer" containerID="2a9443d3d04def9a9e5177a0690878603397a25c450bf27c3353f92f3a70658a" Nov 02 08:30:02.756177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.754411 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerStarted","Data":"defed8d782b34c257d41d8557cbde842b9bf6ac931ba115b19fc716061991f6c"} Nov 02 08:30:02.756177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.754761 5055 scope.go:122] "RemoveContainer" containerID="defed8d782b34c257d41d8557cbde842b9bf6ac931ba115b19fc716061991f6c" Nov 02 08:30:02.758450 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.758410 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerStarted","Data":"40bdfdf7e080a07eeb2a3fab3dd5e843d5dcddb9bd1436182b2b1ba138c68921"} Nov 02 08:30:02.760663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.760631 5055 generic.go:356] "Generic (PLEG): container finished" podID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerID="b8f01ec8b51fa31d307d53d6cb1824484f1b28c16cb8b4d2407b8e76efb7dc7d" exitCode=60 Nov 02 08:30:02.760755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.760736 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerDied","Data":"b8f01ec8b51fa31d307d53d6cb1824484f1b28c16cb8b4d2407b8e76efb7dc7d"} Nov 02 08:30:02.761124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.761104 5055 scope.go:122] "RemoveContainer" containerID="b8f01ec8b51fa31d307d53d6cb1824484f1b28c16cb8b4d2407b8e76efb7dc7d" Nov 02 08:30:02.768868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.768841 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerID="0b59652c4f64a996852d6fb93201a943bb60c24a6b4031e218a208b0996d4dc1" exitCode=60 Nov 02 08:30:02.768945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.768911 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerDied","Data":"0b59652c4f64a996852d6fb93201a943bb60c24a6b4031e218a208b0996d4dc1"} Nov 02 08:30:02.769270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.769249 5055 scope.go:122] "RemoveContainer" containerID="0b59652c4f64a996852d6fb93201a943bb60c24a6b4031e218a208b0996d4dc1" Nov 02 08:30:02.778216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.777853 5055 generic.go:356] "Generic (PLEG): container finished" podID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerID="8664d6a844cbd32eb8ce023bf0bc6ad111dc6f9984362067c30f9e24924db508" exitCode=60 Nov 02 08:30:02.778216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.777964 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" event={"ID":"240de6f6-cb10-437e-a9dd-122ba69b21b3","Type":"ContainerDied","Data":"8664d6a844cbd32eb8ce023bf0bc6ad111dc6f9984362067c30f9e24924db508"} Nov 02 08:30:02.778837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:02.778805 5055 scope.go:122] "RemoveContainer" containerID="8664d6a844cbd32eb8ce023bf0bc6ad111dc6f9984362067c30f9e24924db508" Nov 02 08:30:03.055404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.055290 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:30:03.055404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.055353 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:03.512695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.511852 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" podStartSLOduration=3.511836652 podStartE2EDuration="3.511836652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:03.511422579 +0000 UTC m=+1450.602042999" watchObservedRunningTime="2026-11-02 08:30:03.511836652 +0000 UTC m=+1450.602457071" Nov 02 08:30:03.801122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.801037 5055 generic.go:356] "Generic (PLEG): container finished" podID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerID="adbdc00102a115bbab795b3a1b4905ece8fe4a7f4961b973e9054a158b2b5ad8" exitCode=60 Nov 02 08:30:03.801865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.801835 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerDied","Data":"adbdc00102a115bbab795b3a1b4905ece8fe4a7f4961b973e9054a158b2b5ad8"} Nov 02 08:30:03.802065 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.802030 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerStarted","Data":"ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174"} Nov 02 08:30:03.805296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.805268 5055 generic.go:356] "Generic (PLEG): container finished" podID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerID="2a9443d3d04def9a9e5177a0690878603397a25c450bf27c3353f92f3a70658a" exitCode=60 Nov 02 08:30:03.805528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.805504 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerDied","Data":"2a9443d3d04def9a9e5177a0690878603397a25c450bf27c3353f92f3a70658a"} Nov 02 08:30:03.805675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.805657 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerStarted","Data":"752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d"} Nov 02 08:30:03.809191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.809162 5055 generic.go:356] "Generic (PLEG): container finished" podID="88030485-a961-4f5d-bab9-f592f564af2d" containerID="defed8d782b34c257d41d8557cbde842b9bf6ac931ba115b19fc716061991f6c" exitCode=60 Nov 02 08:30:03.809411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.809329 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerDied","Data":"defed8d782b34c257d41d8557cbde842b9bf6ac931ba115b19fc716061991f6c"} Nov 02 08:30:03.809411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.809382 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerStarted","Data":"7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676"} Nov 02 08:30:03.811737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.811706 5055 generic.go:356] "Generic (PLEG): container finished" podID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerID="40bdfdf7e080a07eeb2a3fab3dd5e843d5dcddb9bd1436182b2b1ba138c68921" exitCode=60 Nov 02 08:30:03.811893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.811876 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerDied","Data":"40bdfdf7e080a07eeb2a3fab3dd5e843d5dcddb9bd1436182b2b1ba138c68921"} Nov 02 08:30:03.812391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.812369 5055 scope.go:122] "RemoveContainer" containerID="40bdfdf7e080a07eeb2a3fab3dd5e843d5dcddb9bd1436182b2b1ba138c68921" Nov 02 08:30:03.817354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.817319 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerStarted","Data":"11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91"} Nov 02 08:30:03.824651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.823670 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerStarted","Data":"77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674"} Nov 02 08:30:03.827439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.827221 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" event={"ID":"240de6f6-cb10-437e-a9dd-122ba69b21b3","Type":"ContainerStarted","Data":"5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2"} Nov 02 08:30:03.838347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.838311 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerStarted","Data":"2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2"} Nov 02 08:30:03.915940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:03.915866 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" podStartSLOduration=3.915848136 podStartE2EDuration="3.915848136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:03.911903611 +0000 UTC m=+1451.002524053" watchObservedRunningTime="2026-11-02 08:30:03.915848136 +0000 UTC m=+1451.006468556" Nov 02 08:30:04.028916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.027007 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" podStartSLOduration=4.026975747 podStartE2EDuration="4.026975747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.025294782 +0000 UTC m=+1451.115915202" watchObservedRunningTime="2026-11-02 08:30:04.026975747 +0000 UTC m=+1451.117596166" Nov 02 08:30:04.137965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.137893 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" podStartSLOduration=4.13787083 podStartE2EDuration="4.13787083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.132623115 +0000 UTC m=+1451.223243571" watchObservedRunningTime="2026-11-02 08:30:04.13787083 +0000 UTC m=+1451.228491250" Nov 02 08:30:04.367488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.359745 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" podStartSLOduration=4.359724824 podStartE2EDuration="4.359724824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.356269882 +0000 UTC m=+1451.446890302" watchObservedRunningTime="2026-11-02 08:30:04.359724824 +0000 UTC m=+1451.450345244" Nov 02 08:30:04.588532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.588463 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" podStartSLOduration=4.588443356 podStartE2EDuration="4.588443356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.579123074 +0000 UTC m=+1451.669743494" watchObservedRunningTime="2026-11-02 08:30:04.588443356 +0000 UTC m=+1451.679063777" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.861910 5055 generic.go:356] "Generic (PLEG): container finished" podID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerID="11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.862010 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerDied","Data":"11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.862274 5055 scope.go:122] "RemoveContainer" containerID="b8f01ec8b51fa31d307d53d6cb1824484f1b28c16cb8b4d2407b8e76efb7dc7d" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.864077 5055 scope.go:122] "RemoveContainer" containerID="11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.864451 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8_odh-ai-gateway-infra(997c95e9-13d4-4528-b755-d7b070a1e15f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.864586 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerID="77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.864687 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerDied","Data":"77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.865008 5055 scope.go:122] "RemoveContainer" containerID="77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.865353 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl_odh-ai-gateway-infra(0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.866909 5055 generic.go:356] "Generic (PLEG): container finished" podID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerID="5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.866963 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" event={"ID":"240de6f6-cb10-437e-a9dd-122ba69b21b3","Type":"ContainerDied","Data":"5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.867213 5055 scope.go:122] "RemoveContainer" containerID="5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.867513 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg_odh-ai-gateway-infra(240de6f6-cb10-437e-a9dd-122ba69b21b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.868912 5055 generic.go:356] "Generic (PLEG): container finished" podID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerID="2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.868991 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerDied","Data":"2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.869182 5055 scope.go:122] "RemoveContainer" containerID="2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.869429 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2_odh-ai-gateway-infra(afadb26d-c33d-4dd0-905c-e2542d9a3806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.871374 5055 generic.go:356] "Generic (PLEG): container finished" podID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerID="ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.871437 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerDied","Data":"ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.871616 5055 scope.go:122] "RemoveContainer" containerID="ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.871856 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr_odh-ai-gateway-infra(23d0eda2-617d-41f0-a62f-8ea2e2b5300c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.873509 5055 generic.go:356] "Generic (PLEG): container finished" podID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerID="752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.873567 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerDied","Data":"752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.873793 5055 scope.go:122] "RemoveContainer" containerID="752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.874019 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr_odh-ai-gateway-infra(71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.875572 5055 generic.go:356] "Generic (PLEG): container finished" podID="88030485-a961-4f5d-bab9-f592f564af2d" containerID="7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.875636 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerDied","Data":"7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.875901 5055 scope.go:122] "RemoveContainer" containerID="7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.876198 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd_odh-ai-gateway-infra(88030485-a961-4f5d-bab9-f592f564af2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" podUID="88030485-a961-4f5d-bab9-f592f564af2d" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.877445 5055 generic.go:356] "Generic (PLEG): container finished" podID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerID="b90131f77f0621a194c886b3917aad746b7dd26b3927c3dc1a34a314aeb059b5" exitCode=60 Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.877469 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerDied","Data":"b90131f77f0621a194c886b3917aad746b7dd26b3927c3dc1a34a314aeb059b5"} Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.877939 5055 scope.go:122] "RemoveContainer" containerID="b90131f77f0621a194c886b3917aad746b7dd26b3927c3dc1a34a314aeb059b5" Nov 02 08:30:04.902938 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:04.878165 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz_odh-ai-gateway-infra(e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" Nov 02 08:30:04.932072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.932052 5055 scope.go:122] "RemoveContainer" containerID="0b59652c4f64a996852d6fb93201a943bb60c24a6b4031e218a208b0996d4dc1" Nov 02 08:30:04.970876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:04.970776 5055 scope.go:122] "RemoveContainer" containerID="8664d6a844cbd32eb8ce023bf0bc6ad111dc6f9984362067c30f9e24924db508" Nov 02 08:30:05.006709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.006675 5055 scope.go:122] "RemoveContainer" containerID="c7f3e36589e5eba33686630bbd6adb52ee349896cba35a658affa6d60613f8ee" Nov 02 08:30:05.055692 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.055664 5055 scope.go:122] "RemoveContainer" containerID="adbdc00102a115bbab795b3a1b4905ece8fe4a7f4961b973e9054a158b2b5ad8" Nov 02 08:30:05.097356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.097320 5055 scope.go:122] "RemoveContainer" containerID="2a9443d3d04def9a9e5177a0690878603397a25c450bf27c3353f92f3a70658a" Nov 02 08:30:05.136900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.136658 5055 scope.go:122] "RemoveContainer" containerID="defed8d782b34c257d41d8557cbde842b9bf6ac931ba115b19fc716061991f6c" Nov 02 08:30:05.187691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.187657 5055 scope.go:122] "RemoveContainer" containerID="40bdfdf7e080a07eeb2a3fab3dd5e843d5dcddb9bd1436182b2b1ba138c68921" Nov 02 08:30:05.892645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.892619 5055 scope.go:122] "RemoveContainer" containerID="11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91" Nov 02 08:30:05.893335 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.893299 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8_odh-ai-gateway-infra(997c95e9-13d4-4528-b755-d7b070a1e15f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" Nov 02 08:30:05.894663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.894639 5055 scope.go:122] "RemoveContainer" containerID="77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674" Nov 02 08:30:05.895005 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.894980 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl_odh-ai-gateway-infra(0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" Nov 02 08:30:05.896688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.896671 5055 scope.go:122] "RemoveContainer" containerID="5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2" Nov 02 08:30:05.896999 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.896970 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg_odh-ai-gateway-infra(240de6f6-cb10-437e-a9dd-122ba69b21b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" Nov 02 08:30:05.898729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.898713 5055 scope.go:122] "RemoveContainer" containerID="2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2" Nov 02 08:30:05.899007 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.898983 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2_odh-ai-gateway-infra(afadb26d-c33d-4dd0-905c-e2542d9a3806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" Nov 02 08:30:05.900514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.900487 5055 generic.go:356] "Generic (PLEG): container finished" podID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerID="24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59" exitCode=0 Nov 02 08:30:05.900619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.900552 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" event={"ID":"765bdaab-c78e-4491-83f1-2c0bc4f4d510","Type":"ContainerDied","Data":"24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59"} Nov 02 08:30:05.900971 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.900952 5055 scope.go:122] "RemoveContainer" containerID="24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59" Nov 02 08:30:05.902586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.902550 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:05.902674 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.902624 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:05.902674 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.902660 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:05.902933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.902917 5055 scope.go:122] "RemoveContainer" containerID="ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174" Nov 02 08:30:05.903222 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.903200 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr_odh-ai-gateway-infra(23d0eda2-617d-41f0-a62f-8ea2e2b5300c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" Nov 02 08:30:05.905516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.905496 5055 scope.go:122] "RemoveContainer" containerID="752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d" Nov 02 08:30:05.905737 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.905717 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr_odh-ai-gateway-infra(71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" Nov 02 08:30:05.907078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.907058 5055 scope.go:122] "RemoveContainer" containerID="7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676" Nov 02 08:30:05.907334 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.907310 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd_odh-ai-gateway-infra(88030485-a961-4f5d-bab9-f592f564af2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" podUID="88030485-a961-4f5d-bab9-f592f564af2d" Nov 02 08:30:05.909308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:05.909289 5055 scope.go:122] "RemoveContainer" containerID="b90131f77f0621a194c886b3917aad746b7dd26b3927c3dc1a34a314aeb059b5" Nov 02 08:30:05.909546 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:05.909527 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz_odh-ai-gateway-infra(e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" Nov 02 08:30:06.855023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.853114 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d"] Nov 02 08:30:06.865191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.862516 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.904164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.896243 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.941887 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.941974 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942084 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942135 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942240 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942282 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942394 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942430 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.944034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.942498 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhszs\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:06.950207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.949273 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d"] Nov 02 08:30:06.961963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.961900 5055 generic.go:356] "Generic (PLEG): container finished" podID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerID="717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c" exitCode=0 Nov 02 08:30:06.962588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.962539 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" event={"ID":"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a","Type":"ContainerDied","Data":"717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c"} Nov 02 08:30:06.964794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.964397 5055 scope.go:122] "RemoveContainer" containerID="717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c" Nov 02 08:30:06.968084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.968035 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:06.968244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.968215 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" event={"ID":"765bdaab-c78e-4491-83f1-2c0bc4f4d510","Type":"ContainerStarted","Data":"838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7"} Nov 02 08:30:06.968463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.968430 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:06.968685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:06.968655 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:07.043823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.043791 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.043987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.043968 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.044278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.044257 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.044431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.044411 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.044533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.044506 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.044637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.044620 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.044806 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.044786 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhszs\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.044909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.044884 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.045040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.045023 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.045208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.045191 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.045503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.045483 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.045652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.045636 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.046133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.046109 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.046572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.046543 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.048179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.047357 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.049642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.049617 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.117295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.117215 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhszs\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.118829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.118792 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.178578 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.178495 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:07.626356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.626254 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:30:07.631205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.628957 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 08:30:07.631205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.629015 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 08:30:07.723595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.723550 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d"] Nov 02 08:30:07.724690 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:07.724644 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff52f76f_736a_4fb7_9500_186ffbaa1063.slice/crio-625b48f2dbbbae212c8fe13cc7f4f1affb5f630db875ad17c5e1c7bdecbc5153 WatchSource:0}: Error finding container 625b48f2dbbbae212c8fe13cc7f4f1affb5f630db875ad17c5e1c7bdecbc5153: Status 404 returned error can't find the container with id 625b48f2dbbbae212c8fe13cc7f4f1affb5f630db875ad17c5e1c7bdecbc5153 Nov 02 08:30:07.728393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.728356 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:07.728505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.728445 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:07.728505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.728491 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:07.979210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.979172 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" event={"ID":"ff52f76f-736a-4fb7-9500-186ffbaa1063","Type":"ContainerStarted","Data":"399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a"} Nov 02 08:30:07.979700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.979232 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" event={"ID":"ff52f76f-736a-4fb7-9500-186ffbaa1063","Type":"ContainerStarted","Data":"625b48f2dbbbae212c8fe13cc7f4f1affb5f630db875ad17c5e1c7bdecbc5153"} Nov 02 08:30:07.997561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:07.997520 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" event={"ID":"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a","Type":"ContainerStarted","Data":"289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f"} Nov 02 08:30:08.121170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.110367 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podStartSLOduration=2.110341969 podStartE2EDuration="2.110341969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:08.090949821 +0000 UTC m=+1455.181570265" watchObservedRunningTime="2026-11-02 08:30:08.110341969 +0000 UTC m=+1455.200962390" Nov 02 08:30:08.182209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.179772 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:08.182209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.181775 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:08.182209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.181856 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:08.444588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.444504 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:30:08.444904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.444817 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:08.445040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.445021 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:08.616690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.616653 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 08:30:08.616961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:08.616932 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 08:30:09.179179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:09.178864 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:09.179179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:09.178916 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:09.452244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:09.444617 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:09.452244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:09.444684 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:09.601884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:09.601262 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 08:30:09.601884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:09.601340 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 08:30:10.247698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.247652 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:10.450327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.450274 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:10.490304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.452859 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:10.713905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.713858 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:10.715312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.467072 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw"] Nov 02 08:30:10.735405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.728403 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 08:30:10.735405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.728486 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 08:30:10.867953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:10.867794 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.083347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.083245 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.296236 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.254876 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:11.497714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.497670 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:11.553972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.444370 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.458178 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" podUID="5d13bf2a-d174-4b83-b363-ed42f963cff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.458219 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.458467 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.414654 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570239 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570324 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570399 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h878p\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570528 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570629 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570727 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.570785 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.475020 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:11.599174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.571560 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:11.624399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.624353 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 08:30:11.634173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.628218 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 08:30:11.673168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.670282 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": context deadline exceeded" Nov 02 08:30:11.673168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.672837 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.672976 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673012 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673059 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h878p\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673128 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673206 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673259 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673300 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.673649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.673404 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.674439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.674385 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.674936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.674900 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.676051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.676009 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.676764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.676729 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.735165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.695929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.735822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.735780 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.740398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.740370 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw"] Nov 02 08:30:11.754908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.754828 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.765322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.765278 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:11.789279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.789225 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:11.813304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.805038 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h878p\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.813617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.813575 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:11.816768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.816731 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 08:30:11.836266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.836219 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 08:30:11.892178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.876126 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:11.929742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.918801 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4"] Nov 02 08:30:11.952771 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.952732 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:11.974348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:11.974317 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4"] Nov 02 08:30:12.010754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.010642 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-00cc0e-openshift-default-dockercfg-pfrh6\"" Nov 02 08:30:12.036198 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.026878 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038133 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038205 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038358 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b254h\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038415 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038444 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038486 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038564 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038648 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.041760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.038700 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.154239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.154188 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.154644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.154619 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.154816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.154798 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.155443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.155418 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.162162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.155719 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.162796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.162773 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b254h\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.163125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.161124 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.167583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.160876 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.167736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.167078 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.168017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.167997 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.171381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.171354 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.172625 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.171886 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.173224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.173200 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.198925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.198890 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:12.199172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.199127 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:12.214166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.200910 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.314110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.265525 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.314110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.271463 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.314110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.296591 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b254h\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.314110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.297103 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token\") pod \"e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.379748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.379429 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:12.478130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.468305 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:12.505191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.480322 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:12.505191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.480383 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:12.608180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.596584 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 08:30:12.608180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.596652 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 08:30:12.633161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.633055 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:12.673422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.673334 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 02 08:30:12.858366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.843299 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:12.858366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.856830 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 08:30:12.868107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.868071 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:30:12.871970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.868388 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:30:12.946169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.938511 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:30:12.946169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.939591 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:30:12.967166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.948954 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:30:12.967166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:12.953805 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:30:13.196355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.196288 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:13.237333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.237267 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:13.373084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.373031 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:30:13.457660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.457624 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:13.457907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.457883 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:13.470891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.470856 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw"] Nov 02 08:30:13.510167 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:13.506254 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882efdab_68b3_4b74_8a06_4760a14a669d.slice/crio-998bf55e7db7bd5389ee35c99071de65b31575355cb56491c90b7dc57293191d WatchSource:0}: Error finding container 998bf55e7db7bd5389ee35c99071de65b31575355cb56491c90b7dc57293191d: Status 404 returned error can't find the container with id 998bf55e7db7bd5389ee35c99071de65b31575355cb56491c90b7dc57293191d Nov 02 08:30:13.547014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.546959 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:13.547187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.547075 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:13.547187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.547133 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:13.654191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.652696 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:30:13.657459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.654396 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:30:13.658360 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:13.658332 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:30:14.072178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.064107 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:14.072563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.072524 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:14.072840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.072816 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:14.217170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.210024 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:14.217170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.210093 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:14.255946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.255154 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr"] Nov 02 08:30:14.435630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.428336 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4"] Nov 02 08:30:14.435630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.428519 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.449061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.446948 5055 patch_prober.go:29] interesting pod/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 08:30:14.449061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.447028 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 08:30:14.449061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.447948 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr"] Nov 02 08:30:14.482682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.481496 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" event={"ID":"882efdab-68b3-4b74-8a06-4760a14a669d","Type":"ContainerStarted","Data":"d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09"} Nov 02 08:30:14.482682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.481579 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" event={"ID":"882efdab-68b3-4b74-8a06-4760a14a669d","Type":"ContainerStarted","Data":"998bf55e7db7bd5389ee35c99071de65b31575355cb56491c90b7dc57293191d"} Nov 02 08:30:14.491690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.491644 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-19ee94-openshift-default-dockercfg-m6th5\"" Nov 02 08:30:14.497192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.497150 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" event={"ID":"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a","Type":"ContainerStarted","Data":"6bd119c9f4794d97ecabc0392ab329ed22ebea4defbd5d48610dc699148eeb66"} Nov 02 08:30:14.507435 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.507397 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.509334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.509297 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.509653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.509624 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.509860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.509839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.510006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.509989 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.510182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.510165 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.510358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.510338 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.510613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.510596 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.510854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.510836 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxwx\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617544 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxwx\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617638 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617671 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617747 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617814 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617886 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617935 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.617992 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.618241 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.619565 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.620271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.635824 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.637058 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.641225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.637351 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.646273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.646245 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.655522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.655479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.673463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.673352 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.675491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.675403 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podStartSLOduration=4.675382618 podStartE2EDuration="4.675382618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:14.665647937 +0000 UTC m=+1461.756268358" watchObservedRunningTime="2026-11-02 08:30:14.675382618 +0000 UTC m=+1461.766003038" Nov 02 08:30:14.739587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.739550 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxwx\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx\") pod \"e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:14.834618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:14.834561 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:15.030277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.030239 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:15.032046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.031119 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:15.032046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.031217 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:15.122972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.122936 5055 scope.go:122] "RemoveContainer" containerID="752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d" Nov 02 08:30:15.178901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.178861 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:15.179056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.178916 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:15.315855 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.315417 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr"] Nov 02 08:30:15.347321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.347276 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:15.347563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.347535 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:15.347709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.347689 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:15.449614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.449582 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:30:15.450275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.450240 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:30:15.451278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.451251 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:30:15.555197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.555106 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" event={"ID":"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a","Type":"ContainerStarted","Data":"42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2"} Nov 02 08:30:15.557323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.557293 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" event={"ID":"0e503d1b-7474-4901-94ac-978681e017cc","Type":"ContainerStarted","Data":"8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3"} Nov 02 08:30:15.557482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.557348 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" event={"ID":"0e503d1b-7474-4901-94ac-978681e017cc","Type":"ContainerStarted","Data":"f9a934af8fbdcc865e1e48b10eb4f5d7f8f51437a944b723c2d3c4e470894373"} Nov 02 08:30:15.740650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.740590 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podStartSLOduration=2.740577105 podStartE2EDuration="2.740577105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:15.736387565 +0000 UTC m=+1462.827007986" watchObservedRunningTime="2026-11-02 08:30:15.740577105 +0000 UTC m=+1462.831197543" Nov 02 08:30:15.835303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:15.835261 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:16.045865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.045821 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:16.046089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.045883 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:16.046089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.045996 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podStartSLOduration=6.045978216 podStartE2EDuration="6.045978216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:16.006670854 +0000 UTC m=+1463.097291274" watchObservedRunningTime="2026-11-02 08:30:16.045978216 +0000 UTC m=+1463.136598632" Nov 02 08:30:16.086164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.084948 5055 scope.go:122] "RemoveContainer" containerID="5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2" Nov 02 08:30:16.086164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.085620 5055 scope.go:122] "RemoveContainer" containerID="11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91" Nov 02 08:30:16.086164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.085890 5055 scope.go:122] "RemoveContainer" containerID="ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174" Nov 02 08:30:16.209210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.199370 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 08:30:16.209210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.199433 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 08:30:16.388184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.386668 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:16.409166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.403356 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:16.614321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.614211 5055 generic.go:356] "Generic (PLEG): container finished" podID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerID="04925f33820dbd38946c009a6b0c13c6972e9e78fe50eb31427655e6e49c0f4e" exitCode=60 Nov 02 08:30:16.645171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.636651 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerDied","Data":"04925f33820dbd38946c009a6b0c13c6972e9e78fe50eb31427655e6e49c0f4e"} Nov 02 08:30:16.645171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.636778 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:16.645171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.636819 5055 scope.go:122] "RemoveContainer" containerID="752ce6baaed96bc86ed0a1541fbb07cdc2fdc6b92277c951606c8b87bd6b9b3d" Nov 02 08:30:16.645171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.638382 5055 scope.go:122] "RemoveContainer" containerID="04925f33820dbd38946c009a6b0c13c6972e9e78fe50eb31427655e6e49c0f4e" Nov 02 08:30:16.645171 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:16.639646 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr_odh-ai-gateway-infra(71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" Nov 02 08:30:16.660516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.660485 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:30:16.878218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:16.878189 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:17.061602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.042303 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:17.061602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.042375 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:17.120167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.120065 5055 scope.go:122] "RemoveContainer" containerID="b90131f77f0621a194c886b3917aad746b7dd26b3927c3dc1a34a314aeb059b5" Nov 02 08:30:17.121416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.121395 5055 scope.go:122] "RemoveContainer" containerID="77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674" Nov 02 08:30:17.650881 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.650841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerStarted","Data":"3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32"} Nov 02 08:30:17.651508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.651485 5055 scope.go:122] "RemoveContainer" containerID="3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32" Nov 02 08:30:17.651978 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:17.651954 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr_odh-ai-gateway-infra(23d0eda2-617d-41f0-a62f-8ea2e2b5300c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" Nov 02 08:30:17.655186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.655130 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerStarted","Data":"fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d"} Nov 02 08:30:17.655607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.655585 5055 scope.go:122] "RemoveContainer" containerID="fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d" Nov 02 08:30:17.656012 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:17.655982 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8_odh-ai-gateway-infra(997c95e9-13d4-4528-b755-d7b070a1e15f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" Nov 02 08:30:17.657512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.657492 5055 generic.go:356] "Generic (PLEG): container finished" podID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerID="a6bad34e5eb96d336a1535fdf7dd05909bdbf82aec485d3ed09db6823644fc86" exitCode=60 Nov 02 08:30:17.657581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.657539 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" event={"ID":"240de6f6-cb10-437e-a9dd-122ba69b21b3","Type":"ContainerDied","Data":"a6bad34e5eb96d336a1535fdf7dd05909bdbf82aec485d3ed09db6823644fc86"} Nov 02 08:30:17.657619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.657599 5055 scope.go:122] "RemoveContainer" containerID="5eda85f2d57e6738150e58e924cb06f08c70aa964f18dcd9c813ddcae203cae2" Nov 02 08:30:17.658220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.657941 5055 scope.go:122] "RemoveContainer" containerID="a6bad34e5eb96d336a1535fdf7dd05909bdbf82aec485d3ed09db6823644fc86" Nov 02 08:30:17.658529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.658466 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:17.658529 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:17.658487 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg_odh-ai-gateway-infra(240de6f6-cb10-437e-a9dd-122ba69b21b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" Nov 02 08:30:17.663246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.659913 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:30:17.729727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.729685 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:17.730721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.730699 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:17.732255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.732030 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:30:17.745623 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:17.745567 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0eda2_617d_41f0_a62f_8ea2e2b5300c.slice/crio-conmon-3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0eda2_617d_41f0_a62f_8ea2e2b5300c.slice/crio-3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997c95e9_13d4_4528_b755_d7b070a1e15f.slice/crio-conmon-fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997c95e9_13d4_4528_b755_d7b070a1e15f.slice/crio-fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240de6f6_cb10_437e_a9dd_122ba69b21b3.slice/crio-conmon-a6bad34e5eb96d336a1535fdf7dd05909bdbf82aec485d3ed09db6823644fc86.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240de6f6_cb10_437e_a9dd_122ba69b21b3.slice/crio-a6bad34e5eb96d336a1535fdf7dd05909bdbf82aec485d3ed09db6823644fc86.scope\": RecentStats: unable to find data in memory cache]" Nov 02 08:30:17.818293 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:17.818240 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr"] Nov 02 08:30:18.034461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.031055 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:18.034461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.031118 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:18.126196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.123781 5055 scope.go:122] "RemoveContainer" containerID="7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676" Nov 02 08:30:18.128559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.128520 5055 scope.go:122] "RemoveContainer" containerID="2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2" Nov 02 08:30:18.429175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.428068 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:18.472120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.472022 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca\") pod \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " Nov 02 08:30:18.472290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.472184 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294\" (UniqueName: \"kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294\") pod \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\" (UID: \"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d\") " Nov 02 08:30:18.472539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.472511 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca" pod "71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" (UID: "71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:18.472872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.472856 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:18.474400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.474374 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294" pod "71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" (UID: "71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d"). InnerVolumeSpecName "kube-api-access-ls294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:18.574739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.574697 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls294\" (UniqueName: \"kubernetes.io/projected/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d-kube-api-access-ls294\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:18.674809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.674768 5055 generic.go:356] "Generic (PLEG): container finished" podID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerID="fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d" exitCode=60 Nov 02 08:30:18.675265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.674839 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerDied","Data":"fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d"} Nov 02 08:30:18.675265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.674912 5055 scope.go:122] "RemoveContainer" containerID="11a7b1d438279fa21cb7d97bc3fb2f8e74d2efaf1558ceed6213b1426c07db91" Nov 02 08:30:18.675426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.675407 5055 scope.go:122] "RemoveContainer" containerID="fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d" Nov 02 08:30:18.675830 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:18.675801 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8_odh-ai-gateway-infra(997c95e9-13d4-4528-b755-d7b070a1e15f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" Nov 02 08:30:18.678332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.678310 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerStarted","Data":"c86de1712186e119d0aaaf0c5dc884b0e8782b104160972b9191426d7aac68fd"} Nov 02 08:30:18.716584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.716548 5055 generic.go:356] "Generic (PLEG): container finished" podID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerID="3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32" exitCode=60 Nov 02 08:30:18.716742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.716619 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerDied","Data":"3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32"} Nov 02 08:30:18.717436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.717410 5055 scope.go:122] "RemoveContainer" containerID="3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32" Nov 02 08:30:18.718006 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:18.717955 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr_odh-ai-gateway-infra(23d0eda2-617d-41f0-a62f-8ea2e2b5300c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" Nov 02 08:30:18.718525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.718498 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" Nov 02 08:30:18.718820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.718537 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr" event={"ID":"71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d","Type":"ContainerDied","Data":"dbf89b97fb144be012ff031a842b0c669a544184b98545a41b74007d7c70678f"} Nov 02 08:30:18.722343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.722272 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerStarted","Data":"9eb8e5f7a21ce007c55268aaee021cf0c1f3b99bd37a25f4a6104a20380bf8eb"} Nov 02 08:30:18.738105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.738014 5055 scope.go:122] "RemoveContainer" containerID="ee314b2f62ed6e73915a5d3ed64104a95f76dadf7d34f8e061dd34f36617b174" Nov 02 08:30:18.753520 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.753496 5055 scope.go:122] "RemoveContainer" containerID="04925f33820dbd38946c009a6b0c13c6972e9e78fe50eb31427655e6e49c0f4e" Nov 02 08:30:18.812172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.811457 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr"] Nov 02 08:30:18.916221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.916169 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr"] Nov 02 08:30:18.921547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:18.921518 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f5bb4f-29893470-t8ktr"] Nov 02 08:30:19.025032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.025000 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8"] Nov 02 08:30:19.028010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.027986 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:19.028075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.028034 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:19.121380 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.120522 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" path="/var/lib/kubelet/pods/71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d/volumes" Nov 02 08:30:19.130089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.130060 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg"] Nov 02 08:30:19.369482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.369445 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:19.414508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.414477 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts\" (UniqueName: \"kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts\") pod \"240de6f6-cb10-437e-a9dd-122ba69b21b3\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " Nov 02 08:30:19.414739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.414580 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca\") pod \"240de6f6-cb10-437e-a9dd-122ba69b21b3\" (UID: \"240de6f6-cb10-437e-a9dd-122ba69b21b3\") " Nov 02 08:30:19.415543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.415521 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca" pod "240de6f6-cb10-437e-a9dd-122ba69b21b3" (UID: "240de6f6-cb10-437e-a9dd-122ba69b21b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:19.417189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.417168 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts" pod "240de6f6-cb10-437e-a9dd-122ba69b21b3" (UID: "240de6f6-cb10-437e-a9dd-122ba69b21b3"). InnerVolumeSpecName "kube-api-access-2dwts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:19.523595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.523552 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwts\" (UniqueName: \"kubernetes.io/projected/240de6f6-cb10-437e-a9dd-122ba69b21b3-kube-api-access-2dwts\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:19.523860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.523837 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240de6f6-cb10-437e-a9dd-122ba69b21b3-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:19.736536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.736500 5055 generic.go:356] "Generic (PLEG): container finished" podID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerID="8bb1e5a113520460de09444c665c9c07ea8e5e7e9609d5d6a2b0581481375087" exitCode=60 Nov 02 08:30:19.737023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.736574 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerDied","Data":"8bb1e5a113520460de09444c665c9c07ea8e5e7e9609d5d6a2b0581481375087"} Nov 02 08:30:19.737023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.736647 5055 scope.go:122] "RemoveContainer" containerID="2399a1062c75b24a9ede01bf9750e37d307a303ed70b133817d31cd8a11767d2" Nov 02 08:30:19.737176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.737135 5055 scope.go:122] "RemoveContainer" containerID="8bb1e5a113520460de09444c665c9c07ea8e5e7e9609d5d6a2b0581481375087" Nov 02 08:30:19.737513 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:19.737490 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2_odh-ai-gateway-infra(afadb26d-c33d-4dd0-905c-e2542d9a3806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" Nov 02 08:30:19.742408 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.742381 5055 generic.go:356] "Generic (PLEG): container finished" podID="88030485-a961-4f5d-bab9-f592f564af2d" containerID="4db012c9eafa40aabcc91db18a836fc26a6fc4d6d20b007d62b291b8bfe57e3a" exitCode=60 Nov 02 08:30:19.742528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.742456 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerDied","Data":"4db012c9eafa40aabcc91db18a836fc26a6fc4d6d20b007d62b291b8bfe57e3a"} Nov 02 08:30:19.742983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.742961 5055 scope.go:122] "RemoveContainer" containerID="4db012c9eafa40aabcc91db18a836fc26a6fc4d6d20b007d62b291b8bfe57e3a" Nov 02 08:30:19.743448 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:19.743423 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd_odh-ai-gateway-infra(88030485-a961-4f5d-bab9-f592f564af2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" podUID="88030485-a961-4f5d-bab9-f592f564af2d" Nov 02 08:30:19.744784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.744765 5055 generic.go:356] "Generic (PLEG): container finished" podID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerID="9eb8e5f7a21ce007c55268aaee021cf0c1f3b99bd37a25f4a6104a20380bf8eb" exitCode=60 Nov 02 08:30:19.744867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.744817 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerDied","Data":"9eb8e5f7a21ce007c55268aaee021cf0c1f3b99bd37a25f4a6104a20380bf8eb"} Nov 02 08:30:19.745293 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.745273 5055 scope.go:122] "RemoveContainer" containerID="9eb8e5f7a21ce007c55268aaee021cf0c1f3b99bd37a25f4a6104a20380bf8eb" Nov 02 08:30:19.745673 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:19.745644 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz_odh-ai-gateway-infra(e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" Nov 02 08:30:19.749537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.749515 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerID="c86de1712186e119d0aaaf0c5dc884b0e8782b104160972b9191426d7aac68fd" exitCode=60 Nov 02 08:30:19.749614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.749576 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerDied","Data":"c86de1712186e119d0aaaf0c5dc884b0e8782b104160972b9191426d7aac68fd"} Nov 02 08:30:19.749899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.749885 5055 scope.go:122] "RemoveContainer" containerID="c86de1712186e119d0aaaf0c5dc884b0e8782b104160972b9191426d7aac68fd" Nov 02 08:30:19.750219 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:19.750200 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl_odh-ai-gateway-infra(0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" Nov 02 08:30:19.752032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.752015 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" Nov 02 08:30:19.752297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.752268 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg" event={"ID":"240de6f6-cb10-437e-a9dd-122ba69b21b3","Type":"ContainerDied","Data":"ee0162b199673b8799c24d5f2929d920f147d3ecc5e3174121d30db1ea931cdc"} Nov 02 08:30:19.754102 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.754082 5055 scope.go:122] "RemoveContainer" containerID="7151d279f824764ebf68564c0c8d8ebc7332de15070dda718792549e13da8676" Nov 02 08:30:19.772539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.772508 5055 scope.go:122] "RemoveContainer" containerID="b90131f77f0621a194c886b3917aad746b7dd26b3927c3dc1a34a314aeb059b5" Nov 02 08:30:19.821793 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.821760 5055 scope.go:122] "RemoveContainer" containerID="77d0a46a88f2033d053bb9db15bfc4b77c515a59c1a30643588e94ffbc2ac674" Nov 02 08:30:19.840213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:19.840183 5055 scope.go:122] "RemoveContainer" containerID="a6bad34e5eb96d336a1535fdf7dd05909bdbf82aec485d3ed09db6823644fc86" Nov 02 08:30:20.022498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.022464 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:20.027817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.027779 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:20.027969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.027836 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:20.061081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.061055 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:20.132225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.132127 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca\") pod \"997c95e9-13d4-4528-b755-d7b070a1e15f\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " Nov 02 08:30:20.132386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.132256 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78\" (UniqueName: \"kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78\") pod \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " Nov 02 08:30:20.132386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.132323 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j\" (UniqueName: \"kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j\") pod \"997c95e9-13d4-4528-b755-d7b070a1e15f\" (UID: \"997c95e9-13d4-4528-b755-d7b070a1e15f\") " Nov 02 08:30:20.132386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.132372 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca\") pod \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\" (UID: \"23d0eda2-617d-41f0-a62f-8ea2e2b5300c\") " Nov 02 08:30:20.133386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.133352 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca" pod "23d0eda2-617d-41f0-a62f-8ea2e2b5300c" (UID: "23d0eda2-617d-41f0-a62f-8ea2e2b5300c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:20.133517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.133357 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca" pod "997c95e9-13d4-4528-b755-d7b070a1e15f" (UID: "997c95e9-13d4-4528-b755-d7b070a1e15f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:20.135634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.135604 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j" pod "997c95e9-13d4-4528-b755-d7b070a1e15f" (UID: "997c95e9-13d4-4528-b755-d7b070a1e15f"). InnerVolumeSpecName "kube-api-access-8td5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:20.135863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.135840 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78" pod "23d0eda2-617d-41f0-a62f-8ea2e2b5300c" (UID: "23d0eda2-617d-41f0-a62f-8ea2e2b5300c"). InnerVolumeSpecName "kube-api-access-67h78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:20.144106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.144076 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz"] Nov 02 08:30:20.234412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.234375 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c95e9-13d4-4528-b755-d7b070a1e15f-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:20.234412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.234406 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67h78\" (UniqueName: \"kubernetes.io/projected/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-kube-api-access-67h78\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:20.234412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.234417 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8td5j\" (UniqueName: \"kubernetes.io/projected/997c95e9-13d4-4528-b755-d7b070a1e15f-kube-api-access-8td5j\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:20.234715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.234426 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0eda2-617d-41f0-a62f-8ea2e2b5300c-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:20.247101 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.247068 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl"] Nov 02 08:30:20.573709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.573670 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg"] Nov 02 08:30:20.615288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.615248 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893470-t5qqg"] Nov 02 08:30:20.767991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.767868 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" event={"ID":"23d0eda2-617d-41f0-a62f-8ea2e2b5300c","Type":"ContainerDied","Data":"c8a53a5b3b0f0c58bd7e2312a37f1f78098d1a476b428a226dabf6ad0117166e"} Nov 02 08:30:20.768430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.767992 5055 scope.go:122] "RemoveContainer" containerID="3f86991cf2eb14eee4bea173afbf5e0bbcf52c1e6e191b7a1f78a27fb9448a32" Nov 02 08:30:20.768430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.768270 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr" Nov 02 08:30:20.775265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.775236 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" event={"ID":"997c95e9-13d4-4528-b755-d7b070a1e15f","Type":"ContainerDied","Data":"2619b4ed6fdcced24a7d92800aa8e6a930915fc93ea5cfe9d57b643a40192fba"} Nov 02 08:30:20.775372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.775357 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8" Nov 02 08:30:20.840393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.839441 5055 scope.go:122] "RemoveContainer" containerID="fec36d13a495117e66bfd8fc1e252ad7818a3e6a72e33eb5f9460bbd65eed02d" Nov 02 08:30:20.912936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:20.912862 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2"] Nov 02 08:30:21.027801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.027710 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:21.027962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.027794 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:21.103276 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.103234 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" path="/var/lib/kubelet/pods/240de6f6-cb10-437e-a9dd-122ba69b21b3/volumes" Nov 02 08:30:21.127783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.127702 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:21.148624 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.148595 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk\" (UniqueName: \"kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk\") pod \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " Nov 02 08:30:21.149018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.148999 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca\") pod \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\" (UID: \"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c\") " Nov 02 08:30:21.150773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.150755 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca" pod "0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" (UID: "0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:21.153316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.153285 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk" pod "0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" (UID: "0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c"). InnerVolumeSpecName "kube-api-access-8b9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:21.198869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.198825 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:21.209510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.209483 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:21.250723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.250671 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4\" (UniqueName: \"kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4\") pod \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " Nov 02 08:30:21.250873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.250799 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca\") pod \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\" (UID: \"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd\") " Nov 02 08:30:21.250954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.250928 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca\") pod \"afadb26d-c33d-4dd0-905c-e2542d9a3806\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " Nov 02 08:30:21.251041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251017 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz\" (UniqueName: \"kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz\") pod \"afadb26d-c33d-4dd0-905c-e2542d9a3806\" (UID: \"afadb26d-c33d-4dd0-905c-e2542d9a3806\") " Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251375 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca" pod "e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" (UID: "e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251636 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca" pod "afadb26d-c33d-4dd0-905c-e2542d9a3806" (UID: "afadb26d-c33d-4dd0-905c-e2542d9a3806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251763 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd"] Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251857 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afadb26d-c33d-4dd0-905c-e2542d9a3806-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251879 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251894 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9tk\" (UniqueName: \"kubernetes.io/projected/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c-kube-api-access-8b9tk\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.251939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.251910 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.253745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.253715 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz" pod "afadb26d-c33d-4dd0-905c-e2542d9a3806" (UID: "afadb26d-c33d-4dd0-905c-e2542d9a3806"). InnerVolumeSpecName "kube-api-access-vbrlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:21.253871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.253741 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4" pod "e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" (UID: "e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd"). InnerVolumeSpecName "kube-api-access-fq6g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:21.359297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.359263 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrlz\" (UniqueName: \"kubernetes.io/projected/afadb26d-c33d-4dd0-905c-e2542d9a3806-kube-api-access-vbrlz\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.359297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.359295 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6g4\" (UniqueName: \"kubernetes.io/projected/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd-kube-api-access-fq6g4\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.466702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.466652 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:21.535638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.535598 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8"] Nov 02 08:30:21.540896 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.540852 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893470-drws8"] Nov 02 08:30:21.562667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.562442 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8\") pod \"88030485-a961-4f5d-bab9-f592f564af2d\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " Nov 02 08:30:21.562667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.562533 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca\") pod \"88030485-a961-4f5d-bab9-f592f564af2d\" (UID: \"88030485-a961-4f5d-bab9-f592f564af2d\") " Nov 02 08:30:21.563896 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.563857 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca" pod "88030485-a961-4f5d-bab9-f592f564af2d" (UID: "88030485-a961-4f5d-bab9-f592f564af2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:21.566372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.566335 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8" pod "88030485-a961-4f5d-bab9-f592f564af2d" (UID: "88030485-a961-4f5d-bab9-f592f564af2d"). InnerVolumeSpecName "kube-api-access-mqxc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:21.642458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.642424 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr"] Nov 02 08:30:21.647787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.647758 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b26f03-29893470-9n7jr"] Nov 02 08:30:21.665614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.665580 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88030485-a961-4f5d-bab9-f592f564af2d-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.665614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.665613 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/88030485-a961-4f5d-bab9-f592f564af2d-kube-api-access-mqxc8\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:21.865500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.865400 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" Nov 02 08:30:21.865500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.865457 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd" event={"ID":"88030485-a961-4f5d-bab9-f592f564af2d","Type":"ContainerDied","Data":"8147aa64975c588e78e49244d1bf5c0c4ebb66187834a2616d7d2f51c6bdad59"} Nov 02 08:30:21.866078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.865512 5055 scope.go:122] "RemoveContainer" containerID="4db012c9eafa40aabcc91db18a836fc26a6fc4d6d20b007d62b291b8bfe57e3a" Nov 02 08:30:21.867415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.867396 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" Nov 02 08:30:21.867525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.867478 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz" event={"ID":"e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd","Type":"ContainerDied","Data":"edc2d4117268c909bd6f116d82360e48b683cb0ce46d77f39b6a69fdb97bdba2"} Nov 02 08:30:21.892991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.892547 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" event={"ID":"0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c","Type":"ContainerDied","Data":"744b476f98822d10cccfb0eb9aff21191843cf55b9bc390ce1d0f83ba85df5ba"} Nov 02 08:30:21.892991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.892659 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl" Nov 02 08:30:21.898770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.898732 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" event={"ID":"afadb26d-c33d-4dd0-905c-e2542d9a3806","Type":"ContainerDied","Data":"7eed16c127855a0da83d647a23680268ab63698f519594b69e80b115d0ecbc14"} Nov 02 08:30:21.898899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.898843 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2" Nov 02 08:30:21.900308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.900283 5055 scope.go:122] "RemoveContainer" containerID="9eb8e5f7a21ce007c55268aaee021cf0c1f3b99bd37a25f4a6104a20380bf8eb" Nov 02 08:30:21.941441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.941407 5055 scope.go:122] "RemoveContainer" containerID="c86de1712186e119d0aaaf0c5dc884b0e8782b104160972b9191426d7aac68fd" Nov 02 08:30:21.998671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:21.998585 5055 scope.go:122] "RemoveContainer" containerID="8bb1e5a113520460de09444c665c9c07ea8e5e7e9609d5d6a2b0581481375087" Nov 02 08:30:22.028183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.028135 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 08:30:22.028351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.028207 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 08:30:22.138837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.133658 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2"] Nov 02 08:30:22.141892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.141826 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-108cf9-29893470-6fcs2"] Nov 02 08:30:22.396576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.396208 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz"] Nov 02 08:30:22.476163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.473847 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d77b6-29893470-tjlmz"] Nov 02 08:30:22.533161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.533026 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl"] Nov 02 08:30:22.539342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.539308 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e6ca80-29893470-7d5bl"] Nov 02 08:30:22.593269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.593234 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd"] Nov 02 08:30:22.602919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:22.602884 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d0628e-29893470-j4qmd"] Nov 02 08:30:23.033757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.033724 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:23.034334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.034288 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:23.035339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.035312 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:30:23.097134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.097093 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" path="/var/lib/kubelet/pods/0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c/volumes" Nov 02 08:30:23.097841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.097819 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" path="/var/lib/kubelet/pods/23d0eda2-617d-41f0-a62f-8ea2e2b5300c/volumes" Nov 02 08:30:23.098462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.098444 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88030485-a961-4f5d-bab9-f592f564af2d" path="/var/lib/kubelet/pods/88030485-a961-4f5d-bab9-f592f564af2d/volumes" Nov 02 08:30:23.099474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.099454 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" path="/var/lib/kubelet/pods/997c95e9-13d4-4528-b755-d7b070a1e15f/volumes" Nov 02 08:30:23.099906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.099891 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" path="/var/lib/kubelet/pods/afadb26d-c33d-4dd0-905c-e2542d9a3806/volumes" Nov 02 08:30:23.100318 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:23.100305 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" path="/var/lib/kubelet/pods/e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd/volumes" Nov 02 08:30:24.929616 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:24.929572 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:30:35.007423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:35.007333 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:30:35.773538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:35.773500 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp"] Nov 02 08:30:35.773752 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:35.773724 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" containerID="cri-o://0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727" gracePeriod=30 Nov 02 08:30:35.818503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:35.818468 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl"] Nov 02 08:30:35.818912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:35.818879 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" containerID="cri-o://95b3c0922405fb4e8e2dad4027be0fb5a03b5ed147e3b8fabe405975fe34a17e" gracePeriod=30 Nov 02 08:30:37.643741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.643702 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv"] Nov 02 08:30:37.644442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644403 5055 cpu_manager.go:414] "containerMap: removing container" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.644442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644418 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.644442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644429 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:37.644442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644435 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644447 5055 cpu_manager.go:414] "containerMap: removing container" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644456 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644480 5055 cpu_manager.go:414] "containerMap: removing container" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644485 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644497 5055 cpu_manager.go:414] "containerMap: removing container" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644502 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644508 5055 cpu_manager.go:414] "containerMap: removing container" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644512 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644518 5055 cpu_manager.go:414] "containerMap: removing container" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644524 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644537 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644545 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644554 5055 cpu_manager.go:414] "containerMap: removing container" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644559 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644574 5055 cpu_manager.go:414] "containerMap: removing container" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644579 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644591 5055 cpu_manager.go:414] "containerMap: removing container" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644595 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644601 5055 cpu_manager.go:414] "containerMap: removing container" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644605 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644613 5055 cpu_manager.go:414] "containerMap: removing container" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644617 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644625 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644629 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644638 5055 cpu_manager.go:414] "containerMap: removing container" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644642 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644654 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644660 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644680 5055 cpu_manager.go:414] "containerMap: removing container" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:37.644677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644687 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644847 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644862 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644874 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644881 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644891 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644900 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644908 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644917 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644925 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644936 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644946 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644954 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644962 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644970 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644979 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.644989 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.645002 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:37.646430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.645010 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:37.649515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.649489 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.698884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.698836 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c7f7d7-openshift-default-dockercfg-v8zzq\"" Nov 02 08:30:37.751614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.751530 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv"] Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790115 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790258 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790339 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790380 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790495 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790555 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790594 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790723 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.796322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.790761 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sdq\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892222 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892281 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892360 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892392 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sdq\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892425 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892494 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892545 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892570 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892648 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892665 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.892955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892925 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.893106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.892964 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.893465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.893430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.893588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.893491 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.895208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.895111 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.895208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.895124 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.952178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.949955 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sdq\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.952178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.950692 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token\") pod \"e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:37.963751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:37.963715 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:38.694180 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:38.694131 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc7682d_cdbd_4d67_8b92_97849eb33547.slice/crio-3d4f075653a34e98d1687e1f8b1c603e371a8ee31e75e162823762992ee2bfd7 WatchSource:0}: Error finding container 3d4f075653a34e98d1687e1f8b1c603e371a8ee31e75e162823762992ee2bfd7: Status 404 returned error can't find the container with id 3d4f075653a34e98d1687e1f8b1c603e371a8ee31e75e162823762992ee2bfd7 Nov 02 08:30:38.694645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:38.694497 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv"] Nov 02 08:30:38.697580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:38.697533 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:38.697701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:38.697619 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:38.697701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:38.697653 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:39.322500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:39.322464 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" event={"ID":"3fc7682d-cdbd-4d67-8b92-97849eb33547","Type":"ContainerStarted","Data":"416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171"} Nov 02 08:30:39.322684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:39.322516 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" event={"ID":"3fc7682d-cdbd-4d67-8b92-97849eb33547","Type":"ContainerStarted","Data":"3d4f075653a34e98d1687e1f8b1c603e371a8ee31e75e162823762992ee2bfd7"} Nov 02 08:30:39.433937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:39.433878 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podStartSLOduration=2.433860267 podStartE2EDuration="2.433860267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:39.430331515 +0000 UTC m=+1486.520951935" watchObservedRunningTime="2026-11-02 08:30:39.433860267 +0000 UTC m=+1486.524480687" Nov 02 08:30:39.963949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:39.963904 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:39.971541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:39.971515 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:40.149121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:40.149077 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:30:40.149320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:40.149180 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:30:40.331006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:40.330900 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:40.331964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:40.331940 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:30:41.429315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.428882 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:30:41.431345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.431292 5055 generic.go:356] "Generic (PLEG): container finished" podID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerID="95b3c0922405fb4e8e2dad4027be0fb5a03b5ed147e3b8fabe405975fe34a17e" exitCode=0 Nov 02 08:30:41.431540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.431367 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" event={"ID":"e40b64d2-794e-4276-83e7-a68f0526e5e8","Type":"ContainerDied","Data":"95b3c0922405fb4e8e2dad4027be0fb5a03b5ed147e3b8fabe405975fe34a17e"} Nov 02 08:30:41.431540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.431409 5055 scope.go:122] "RemoveContainer" containerID="db700d276bcdbcd86b83a7695c66010528eb7490a3712843f3070af2fdce5636" Nov 02 08:30:41.434395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.434371 5055 generic.go:356] "Generic (PLEG): container finished" podID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerID="0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727" exitCode=0 Nov 02 08:30:41.435567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.435544 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" Nov 02 08:30:41.436269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.436077 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" event={"ID":"ce777614-2403-468d-bb9d-d7c43c5cab89","Type":"ContainerDied","Data":"0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727"} Nov 02 08:30:41.436422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.436407 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp" event={"ID":"ce777614-2403-468d-bb9d-d7c43c5cab89","Type":"ContainerDied","Data":"bb947a95f71b8720abe2c5ee917032d99300674cf7380b70fc31e476665baa4b"} Nov 02 08:30:41.469973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.461777 5055 scope.go:122] "RemoveContainer" containerID="0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727" Nov 02 08:30:41.500510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.500188 5055 scope.go:122] "RemoveContainer" containerID="0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727" Nov 02 08:30:41.501012 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:41.500901 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727\": container with ID starting with 0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727 not found: ID does not exist" containerID="0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727" Nov 02 08:30:41.501012 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.500948 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727"} err="failed to get container status \"0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727\": rpc error: code = NotFound desc = could not find container \"0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727\": container with ID starting with 0734dac5f748d94a0eee78b4d0012033635e22884cf9510cc13c9ec45c4d0727 not found: ID does not exist" Nov 02 08:30:41.516972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.516943 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:30:41.613523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613433 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.613523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613496 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.613760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613543 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.613760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613587 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.613760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613624 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.613760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613678 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.613760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613715 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.613760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613752 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613758 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613805 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613876 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.613988 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614028 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614051 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.614109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614083 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614212 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614269 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614303 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614363 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token\") pod \"ce777614-2403-468d-bb9d-d7c43c5cab89\" (UID: \"ce777614-2403-468d-bb9d-d7c43c5cab89\") " Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614409 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket\") pod \"e40b64d2-794e-4276-83e7-a68f0526e5e8\" (UID: \"e40b64d2-794e-4276-83e7-a68f0526e5e8\") " Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614498 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614699 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614948 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:41.615157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614924 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615314 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b64d2-794e-4276-83e7-a68f0526e5e8-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615338 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615355 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615368 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615382 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615397 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.614967 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:41.615733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.615598 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.616478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.616452 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.617170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.617133 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.621854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.621454 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:30:41.621854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.621514 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:41.621854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.621597 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "kube-api-access-4fl5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:41.621854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.621706 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:41.621854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.621820 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:30:41.622385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.622361 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "kube-api-access-tgwml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:41.622972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.622946 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy" pod "ce777614-2403-468d-bb9d-d7c43c5cab89" (UID: "ce777614-2403-468d-bb9d-d7c43c5cab89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.623215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.623189 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy" pod "e40b64d2-794e-4276-83e7-a68f0526e5e8" (UID: "e40b64d2-794e-4276-83e7-a68f0526e5e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:30:41.717439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717398 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717437 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717450 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717462 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717475 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce777614-2403-468d-bb9d-d7c43c5cab89-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717488 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717501 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717514 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce777614-2403-468d-bb9d-d7c43c5cab89-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717527 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fl5h\" (UniqueName: \"kubernetes.io/projected/ce777614-2403-468d-bb9d-d7c43c5cab89-kube-api-access-4fl5h\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717542 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717556 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b64d2-794e-4276-83e7-a68f0526e5e8-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.717746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.717570 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/e40b64d2-794e-4276-83e7-a68f0526e5e8-kube-api-access-tgwml\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:41.856326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.856282 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp"] Nov 02 08:30:41.864637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:41.864555 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-nd5dp"] Nov 02 08:30:42.444109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:42.444080 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" Nov 02 08:30:42.444629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:42.444087 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl" event={"ID":"e40b64d2-794e-4276-83e7-a68f0526e5e8","Type":"ContainerDied","Data":"14f1287c6bccfd0a164ff9be57adc3760ab45c4f52b8819153d7fe530fdb3268"} Nov 02 08:30:42.444629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:42.444221 5055 scope.go:122] "RemoveContainer" containerID="95b3c0922405fb4e8e2dad4027be0fb5a03b5ed147e3b8fabe405975fe34a17e" Nov 02 08:30:42.606762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:42.606719 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl"] Nov 02 08:30:42.625469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:42.618911 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-9vbxl"] Nov 02 08:30:43.097089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:43.097042 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" path="/var/lib/kubelet/pods/ce777614-2403-468d-bb9d-d7c43c5cab89/volumes" Nov 02 08:30:43.097772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:43.097745 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" path="/var/lib/kubelet/pods/e40b64d2-794e-4276-83e7-a68f0526e5e8/volumes" Nov 02 08:30:45.035187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035151 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr"] Nov 02 08:30:45.035868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035850 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:45.035918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035870 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:45.035918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035882 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" Nov 02 08:30:45.035918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035887 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" Nov 02 08:30:45.035918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035898 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" Nov 02 08:30:45.035918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035904 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035932 5055 cpu_manager.go:414] "containerMap: removing container" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035937 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035946 5055 cpu_manager.go:414] "containerMap: removing container" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035950 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240de6f6-cb10-437e-a9dd-122ba69b21b3" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035959 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035964 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035970 5055 cpu_manager.go:414] "containerMap: removing container" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035974 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035986 5055 cpu_manager.go:414] "containerMap: removing container" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035990 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88030485-a961-4f5d-bab9-f592f564af2d" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.035999 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036004 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036012 5055 cpu_manager.go:414] "containerMap: removing container" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:45.036068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036016 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036110 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cb5b8e-0f00-47b6-aca4-acd0c9ecbf2d" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036122 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eaa325-ffd0-4688-92b5-7c7cdb9d1dfd" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036130 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036175 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40b64d2-794e-4276-83e7-a68f0526e5e8" containerName="istio-proxy" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036184 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0eda2-617d-41f0-a62f-8ea2e2b5300c" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036193 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="997c95e9-13d4-4528-b755-d7b070a1e15f" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036200 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="afadb26d-c33d-4dd0-905c-e2542d9a3806" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036212 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce6934c-5a76-4ff6-a9fe-60adffaa6f4c" containerName="cleanup" Nov 02 08:30:45.036529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.036220 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce777614-2403-468d-bb9d-d7c43c5cab89" containerName="istio-proxy" Nov 02 08:30:45.040985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.040963 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.073643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.073608 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.073790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.073655 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.073790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.073773 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.073873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.073811 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.073926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.073890 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.073992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.073978 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtzd\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.074038 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.074025 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.089692 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.089660 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 08:30:45.089889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.089710 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 08:30:45.089889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.089710 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 08:30:45.090008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.089901 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p8mvq\"" Nov 02 08:30:45.144772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.144736 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr"] Nov 02 08:30:45.175846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.175804 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.176044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.175863 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.176044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.175985 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.176199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.176064 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.176199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.176118 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.176317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.176262 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtzd\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.176373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.176353 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.179303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.179271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.181778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.181754 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.194483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.194443 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.194634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.194544 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.196318 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.196293 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.230364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.230331 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.232116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.232086 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtzd\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p6pr\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.354861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.354772 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:45.775964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.775930 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr"] Nov 02 08:30:45.778549 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:30:45.778513 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad2561a_78e2_4044_8ced_a14cfad60342.slice/crio-5156692234e12d0c378c143ef1e693cc39a9af9db74b682aa3f1335733b14f60 WatchSource:0}: Error finding container 5156692234e12d0c378c143ef1e693cc39a9af9db74b682aa3f1335733b14f60: Status 404 returned error can't find the container with id 5156692234e12d0c378c143ef1e693cc39a9af9db74b682aa3f1335733b14f60 Nov 02 08:30:45.780780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:45.780763 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:30:46.492919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:46.492882 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" event={"ID":"2ad2561a-78e2-4044-8ced-a14cfad60342","Type":"ContainerStarted","Data":"5156692234e12d0c378c143ef1e693cc39a9af9db74b682aa3f1335733b14f60"} Nov 02 08:30:52.910530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:52.910494 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:52.910808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:52.910568 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:30:53.329342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:53.329307 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:30:53.333203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:53.333133 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:30:53.565314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:53.565270 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" event={"ID":"2ad2561a-78e2-4044-8ced-a14cfad60342","Type":"ContainerStarted","Data":"24bb2eefdf91b83d6a66e3b3ef3c49a787f5139bf2395980e99162d812de3dcb"} Nov 02 08:30:53.565468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:53.565347 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:30:53.577816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:53.575772 5055 scope.go:122] "RemoveContainer" containerID="9f9808c419ef650b97cce91098fe39f337cf986cde8c84fa465e9ee800c62348" Nov 02 08:30:53.679701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:53.679636 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" podStartSLOduration=2.551455958 podStartE2EDuration="9.679616338s" totalImagesPullingTime="7.12816038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:53.677397958 +0000 UTC m=+1500.768018374" watchObservedRunningTime="2026-11-02 08:30:53.679616338 +0000 UTC m=+1500.770236758" Nov 02 08:30:54.575807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:54.575769 5055 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5p6pr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:30:54.576291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:54.575833 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:30:54.926673 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:30:54.926586 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:30:57.575111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:30:57.575083 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:31:17.044967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:17.044930 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d"] Nov 02 08:31:17.101859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:17.101817 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82099558-jt57d"] Nov 02 08:31:19.094365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:19.094326 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3059ceff-af81-46fe-b8af-c7bb7f96293c" path="/var/lib/kubelet/pods/3059ceff-af81-46fe-b8af-c7bb7f96293c/volumes" Nov 02 08:31:24.925081 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:31:24.925037 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:31:35.047213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:35.047173 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn"] Nov 02 08:31:35.095696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:35.095665 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a559b8-dhdsn"] Nov 02 08:31:35.713669 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:35.713633 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw"] Nov 02 08:31:35.722671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:35.722645 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:35.911996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:35.911964 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-69a9bb-openshift-default-dockercfg-6vf2g\"" Nov 02 08:31:35.922075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:35.922025 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw"] Nov 02 08:31:36.042973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.042873 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043007 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043038 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043079 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043189 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvwk\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043307 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043349 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043387 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.043684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.043415 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.144461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144421 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.144461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144459 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144489 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144565 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvwk\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144675 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144707 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144732 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144870 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144888 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144976 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.144981 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.145079 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.145162 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.145961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.145929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.158540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.157667 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.159660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.159632 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.200888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.200853 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.201074 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.200969 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvwk\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.339439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.339348 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:36.764022 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:31:36.763988 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd475b9a6_835f_4ac5_a681_fb2346b2976e.slice/crio-700c50e8cb1efce39c639c437628f2e5bbd61ce82bf60bdaea5c68d16bf58868 WatchSource:0}: Error finding container 700c50e8cb1efce39c639c437628f2e5bbd61ce82bf60bdaea5c68d16bf58868: Status 404 returned error can't find the container with id 700c50e8cb1efce39c639c437628f2e5bbd61ce82bf60bdaea5c68d16bf58868 Nov 02 08:31:36.764192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.764134 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw"] Nov 02 08:31:36.766595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.766558 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:31:36.766691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.766630 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:31:36.766691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.766660 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:31:36.974545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.974495 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" event={"ID":"d475b9a6-835f-4ac5-a681-fb2346b2976e","Type":"ContainerStarted","Data":"389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907"} Nov 02 08:31:36.974742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:36.974566 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" event={"ID":"d475b9a6-835f-4ac5-a681-fb2346b2976e","Type":"ContainerStarted","Data":"700c50e8cb1efce39c639c437628f2e5bbd61ce82bf60bdaea5c68d16bf58868"} Nov 02 08:31:37.087641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:37.087507 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podStartSLOduration=2.087487837 podStartE2EDuration="2.087487837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:37.082687392 +0000 UTC m=+1544.173307816" watchObservedRunningTime="2026-11-02 08:31:37.087487837 +0000 UTC m=+1544.178108258" Nov 02 08:31:37.097377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:37.097338 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6f91e0-2360-420b-9d63-ae2c801ded28" path="/var/lib/kubelet/pods/2f6f91e0-2360-420b-9d63-ae2c801ded28/volumes" Nov 02 08:31:37.340366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:37.340287 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:37.354241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:37.354208 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:37.985773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:37.985570 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:37.986679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:37.986641 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:31:43.045939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:43.045903 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r"] Nov 02 08:31:43.097208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:43.097175 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r"] Nov 02 08:31:43.138896 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:31:43.138851 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e07aef76-x869r\" is forbidden: User \"system:node:ip-10-0-139-247.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-247.us-west-2.compute.internal' and this object" podUID="84073ccf-5668-46ed-ba88-67c80ed180ad" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aef76-x869r" Nov 02 08:31:45.098043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:45.097998 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84073ccf-5668-46ed-ba88-67c80ed180ad" path="/var/lib/kubelet/pods/84073ccf-5668-46ed-ba88-67c80ed180ad/volumes" Nov 02 08:31:48.957464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:48.957430 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:31:51.055295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:51.055255 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb"] Nov 02 08:31:51.119917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:51.119876 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66bd1e85-thtfb"] Nov 02 08:31:53.124408 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:53.124365 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740fe1fe-8ef5-47cf-a742-3ee60f946e18" path="/var/lib/kubelet/pods/740fe1fe-8ef5-47cf-a742-3ee60f946e18/volumes" Nov 02 08:31:54.060817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:54.060778 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl"] Nov 02 08:31:54.119091 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:54.119049 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22e4e003-f2mhl"] Nov 02 08:31:54.936918 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:31:54.936875 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:31:55.121585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:31:55.121540 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0763829-af25-4494-98d8-72acf1aeeac9" path="/var/lib/kubelet/pods/f0763829-af25-4494-98d8-72acf1aeeac9/volumes" Nov 02 08:32:01.243581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.243542 5055 generic.go:356] "Generic (PLEG): container finished" podID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerID="42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2" exitCode=0 Nov 02 08:32:01.244201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.243617 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" event={"ID":"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a","Type":"ContainerDied","Data":"42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2"} Nov 02 08:32:01.244306 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.244208 5055 scope.go:122] "RemoveContainer" containerID="42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2" Nov 02 08:32:01.246090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.246055 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:01.246185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.246130 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:01.246231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.246181 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:01.622307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:01.622207 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:32:02.253985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:02.253946 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" event={"ID":"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a","Type":"ContainerStarted","Data":"20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc"} Nov 02 08:32:02.403164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:02.395939 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:32:02.403164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:02.396071 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 08:32:02.403164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:02.396127 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 08:32:03.390167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:03.386526 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 08:32:03.390167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:03.386607 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 08:32:04.397727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:04.397618 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 08:32:04.398382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:04.398346 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 08:32:05.395166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:05.388660 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 08:32:05.395166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:05.388734 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 08:32:06.398167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:06.388520 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 08:32:06.398167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:06.388589 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 08:32:07.389663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:07.389615 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 08:32:07.390016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:07.389983 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 08:32:08.391434 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:08.391397 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:32:08.391885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:08.391839 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:32:08.392770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:08.392749 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:32:10.103976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:10.103934 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f"] Nov 02 08:32:10.128264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:10.128227 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa7e00b-kmb7f"] Nov 02 08:32:11.100633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:11.100593 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e" path="/var/lib/kubelet/pods/0a26b0bd-c8e0-4b4d-8f21-33dad744eb2e/volumes" Nov 02 08:32:19.007399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.007361 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b"] Nov 02 08:32:19.010865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.010842 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.088583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.088539 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6kg\" (UniqueName: \"kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.088826 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.088805 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.088941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.088919 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.110025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.108923 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:32:19.110025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.108971 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:32:19.110025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.109026 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:32:19.110025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.109058 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:32:19.116550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.116515 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b"] Nov 02 08:32:19.190975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.190937 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.191211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.191011 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6kg\" (UniqueName: \"kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.191335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.191318 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.192180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.192106 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.193864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.193822 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.248095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.248058 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6kg\" (UniqueName: \"kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.324031 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.323947 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:19.714779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:19.714747 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b"] Nov 02 08:32:19.714929 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:32:19.714844 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fa51d4_6d34_45c9_877d_0a95054f1331.slice/crio-3877daba9fcaab0ac55ddb1dfb8f3395657484881a92f14682b7bdf069224b0a WatchSource:0}: Error finding container 3877daba9fcaab0ac55ddb1dfb8f3395657484881a92f14682b7bdf069224b0a: Status 404 returned error can't find the container with id 3877daba9fcaab0ac55ddb1dfb8f3395657484881a92f14682b7bdf069224b0a Nov 02 08:32:20.449495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:20.449461 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" event={"ID":"58fa51d4-6d34-45c9-877d-0a95054f1331","Type":"ContainerStarted","Data":"b32b2c8c61bd1cb0c026ee2e4fae0360d7b027d9dcaf6d8fa9ebdbef2f73084d"} Nov 02 08:32:20.450008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:20.449543 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" event={"ID":"58fa51d4-6d34-45c9-877d-0a95054f1331","Type":"ContainerStarted","Data":"3877daba9fcaab0ac55ddb1dfb8f3395657484881a92f14682b7bdf069224b0a"} Nov 02 08:32:20.566677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:20.566606 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" podStartSLOduration=2.566590317 podStartE2EDuration="2.566590317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:20.561408681 +0000 UTC m=+1587.652029099" watchObservedRunningTime="2026-11-02 08:32:20.566590317 +0000 UTC m=+1587.657210789" Nov 02 08:32:21.070150 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:21.070046 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww"] Nov 02 08:32:21.075593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:21.075565 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242e3caa-7txww"] Nov 02 08:32:21.095598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:21.095561 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ed46ba-8dea-4868-a335-147ac8d8a683" path="/var/lib/kubelet/pods/39ed46ba-8dea-4868-a335-147ac8d8a683/volumes" Nov 02 08:32:21.461895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:21.461859 5055 generic.go:356] "Generic (PLEG): container finished" podID="58fa51d4-6d34-45c9-877d-0a95054f1331" containerID="b32b2c8c61bd1cb0c026ee2e4fae0360d7b027d9dcaf6d8fa9ebdbef2f73084d" exitCode=0 Nov 02 08:32:21.462321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:21.461941 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" event={"ID":"58fa51d4-6d34-45c9-877d-0a95054f1331","Type":"ContainerDied","Data":"b32b2c8c61bd1cb0c026ee2e4fae0360d7b027d9dcaf6d8fa9ebdbef2f73084d"} Nov 02 08:32:22.648525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.648497 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:22.756463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.756358 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca\") pod \"58fa51d4-6d34-45c9-877d-0a95054f1331\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " Nov 02 08:32:22.756463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.756440 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg\" (UniqueName: \"kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg\") pod \"58fa51d4-6d34-45c9-877d-0a95054f1331\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " Nov 02 08:32:22.756683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.756570 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca\") pod \"58fa51d4-6d34-45c9-877d-0a95054f1331\" (UID: \"58fa51d4-6d34-45c9-877d-0a95054f1331\") " Nov 02 08:32:22.756825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.756793 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca" pod "58fa51d4-6d34-45c9-877d-0a95054f1331" (UID: "58fa51d4-6d34-45c9-877d-0a95054f1331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:32:22.757556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.757528 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:32:22.759345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.759315 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg" pod "58fa51d4-6d34-45c9-877d-0a95054f1331" (UID: "58fa51d4-6d34-45c9-877d-0a95054f1331"). InnerVolumeSpecName "kube-api-access-fb6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:32:22.759522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.759344 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca" pod "58fa51d4-6d34-45c9-877d-0a95054f1331" (UID: "58fa51d4-6d34-45c9-877d-0a95054f1331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:32:22.859017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.858973 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb6kg\" (UniqueName: \"kubernetes.io/projected/58fa51d4-6d34-45c9-877d-0a95054f1331-kube-api-access-fb6kg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:32:22.859017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:22.859012 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fa51d4-6d34-45c9-877d-0a95054f1331-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:32:23.480854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:23.480787 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" event={"ID":"58fa51d4-6d34-45c9-877d-0a95054f1331","Type":"ContainerDied","Data":"3877daba9fcaab0ac55ddb1dfb8f3395657484881a92f14682b7bdf069224b0a"} Nov 02 08:32:23.481122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:23.480880 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3877daba9fcaab0ac55ddb1dfb8f3395657484881a92f14682b7bdf069224b0a" Nov 02 08:32:23.481122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:23.480810 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b" Nov 02 08:32:24.923642 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:32:24.923605 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:32:28.055215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:28.055174 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7"] Nov 02 08:32:28.101302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:28.101263 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0387b5f-4hdd7"] Nov 02 08:32:29.094351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:29.094304 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45f3872-8609-4e15-b576-557114ef9ae8" path="/var/lib/kubelet/pods/e45f3872-8609-4e15-b576-557114ef9ae8/volumes" Nov 02 08:32:34.831855 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:34.831819 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:32:44.056842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:44.056782 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb"] Nov 02 08:32:44.102609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:44.102553 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59cc315-dtzkb"] Nov 02 08:32:45.095566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:45.095518 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fbaf00-98a2-48da-9b81-f5d1e723f021" path="/var/lib/kubelet/pods/81fbaf00-98a2-48da-9b81-f5d1e723f021/volumes" Nov 02 08:32:48.701569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.701165 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:32:48.701569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.701249 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:32:48.848542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.835541 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf"] Nov 02 08:32:48.848542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.836604 5055 cpu_manager.go:414] "containerMap: removing container" podUID="58fa51d4-6d34-45c9-877d-0a95054f1331" containerName="revoke-keys" Nov 02 08:32:48.848542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.836626 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fa51d4-6d34-45c9-877d-0a95054f1331" containerName="revoke-keys" Nov 02 08:32:48.848542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.836995 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fa51d4-6d34-45c9-877d-0a95054f1331" containerName="revoke-keys" Nov 02 08:32:48.855025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.854624 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:48.885168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.885116 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf"] Nov 02 08:32:48.965264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:48.951992 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b9eea3-openshift-default-dockercfg-lt5rj\"" Nov 02 08:32:49.039374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039290 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.039727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039459 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.039727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039577 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.039727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039677 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.039727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039714 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.040261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039770 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzk4j\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.040261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039826 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.040261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039873 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.040261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.039915 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141031 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.140990 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141045 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141081 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141171 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141432 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141528 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141560 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141541 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141628 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.141800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.141687 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzk4j\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.142065 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.142032 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.142204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.142112 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.142501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.142470 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.143594 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.143568 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.144466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.144437 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.208113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.207794 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzk4j\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.243513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.225709 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.546780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.515185 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:49.874418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.874330 5055 generic.go:356] "Generic (PLEG): container finished" podID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerID="416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171" exitCode=0 Nov 02 08:32:49.874418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.874403 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" event={"ID":"3fc7682d-cdbd-4d67-8b92-97849eb33547","Type":"ContainerDied","Data":"416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171"} Nov 02 08:32:49.875034 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.875014 5055 scope.go:122] "RemoveContainer" containerID="416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171" Nov 02 08:32:49.876316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.876297 5055 generic.go:356] "Generic (PLEG): container finished" podID="0e503d1b-7474-4901-94ac-978681e017cc" containerID="8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3" exitCode=0 Nov 02 08:32:49.876448 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.876376 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" event={"ID":"0e503d1b-7474-4901-94ac-978681e017cc","Type":"ContainerDied","Data":"8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3"} Nov 02 08:32:49.876788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.876769 5055 scope.go:122] "RemoveContainer" containerID="8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3" Nov 02 08:32:49.876881 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.876820 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.876937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.876885 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.876986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.876935 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.878468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.878442 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.878553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.878504 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.878594 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.878551 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.962777 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:32:49.962695 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf374ffb5_7bb8_4663_9afe_f1a528ea2069.slice/crio-60d0af447c1f97ff0116dcbc1f222f1a4493a31a7487502ab2848cc3683c1d1b WatchSource:0}: Error finding container 60d0af447c1f97ff0116dcbc1f222f1a4493a31a7487502ab2848cc3683c1d1b: Status 404 returned error can't find the container with id 60d0af447c1f97ff0116dcbc1f222f1a4493a31a7487502ab2848cc3683c1d1b Nov 02 08:32:49.966004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.965950 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.966130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.966089 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.966199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.966151 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:32:49.977650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:49.977619 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf"] Nov 02 08:32:50.936168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.925753 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" event={"ID":"3fc7682d-cdbd-4d67-8b92-97849eb33547","Type":"ContainerStarted","Data":"28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b"} Nov 02 08:32:50.949068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.949031 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" event={"ID":"0e503d1b-7474-4901-94ac-978681e017cc","Type":"ContainerStarted","Data":"e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452"} Nov 02 08:32:50.951926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.951885 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" event={"ID":"f374ffb5-7bb8-4663-9afe-f1a528ea2069","Type":"ContainerStarted","Data":"b19ce90bee1516120ebeb15cffcc6a2dd9a2b340ed901c8996490bac28a574b2"} Nov 02 08:32:50.952100 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.951934 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" event={"ID":"f374ffb5-7bb8-4663-9afe-f1a528ea2069","Type":"ContainerStarted","Data":"60d0af447c1f97ff0116dcbc1f222f1a4493a31a7487502ab2848cc3683c1d1b"} Nov 02 08:32:50.964090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.964061 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:32:50.965216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.965192 5055 patch_prober.go:29] interesting pod/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:32:50.965321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:50.965240 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 08:32:51.248842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.248417 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podStartSLOduration=3.24839519 podStartE2EDuration="3.24839519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:51.208972561 +0000 UTC m=+1618.299592992" watchObservedRunningTime="2026-11-02 08:32:51.24839519 +0000 UTC m=+1618.339015615" Nov 02 08:32:51.518223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.517704 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:51.548329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.548293 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:51.835890 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.835778 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 08:32:51.836245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.836195 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 08:32:51.836498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.836481 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:32:51.963425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.962662 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:51.963905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.963880 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:32:51.964010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.963987 5055 patch_prober.go:29] interesting pod/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:32:51.964049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:51.964031 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 08:32:52.842164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:52.835555 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 08:32:52.842164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:52.835626 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 08:32:52.979176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:52.972287 5055 patch_prober.go:29] interesting pod/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:32:52.979176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:52.972360 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 08:32:53.318167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.314153 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9"] Nov 02 08:32:53.329214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.329178 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4b070-d4wv9"] Nov 02 08:32:53.760833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.760408 5055 scope.go:122] "RemoveContainer" containerID="6e833ef5a7cd5bd86df6c91f6acf4babee983dea8a694e49c8f794f70f814020" Nov 02 08:32:53.775882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.775853 5055 scope.go:122] "RemoveContainer" containerID="2798c109520437bb8719961e34b1900ff2617dcd6e9f8d13166e59673c1c02fc" Nov 02 08:32:53.858179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.857021 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 08:32:53.858179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.857098 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 08:32:53.948175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.946338 5055 scope.go:122] "RemoveContainer" containerID="663b8af2bd0671be76761ba802d80353a89adb74eec388702610ba412b37d208" Nov 02 08:32:53.964865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.964617 5055 patch_prober.go:29] interesting pod/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:32:53.964865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.964725 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 08:32:53.968499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:53.968470 5055 scope.go:122] "RemoveContainer" containerID="8529463f474fd5f863420fc57074fd3e3cb3f22e6f470d0379517939403e4e29" Nov 02 08:32:54.013745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:54.007937 5055 scope.go:122] "RemoveContainer" containerID="537330bad4206f5338f7e578ef317c5228dfebff33134b06a6b4d17a97ec5787" Nov 02 08:32:54.837166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:54.836868 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 08:32:54.837166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:54.836955 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 08:32:54.964588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:54.964545 5055 patch_prober.go:29] interesting pod/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:32:54.964755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:54.964619 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 08:32:54.970789 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:32:54.970755 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:32:55.157258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:55.157166 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc44d36-68c6-47f0-b967-6147017bc5b0" path="/var/lib/kubelet/pods/6bc44d36-68c6-47f0-b967-6147017bc5b0/volumes" Nov 02 08:32:55.333168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:55.332273 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:32:55.835737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:55.835686 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 08:32:55.835941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:55.835773 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 08:32:55.980115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:55.979618 5055 patch_prober.go:29] interesting pod/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:32:55.980115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:55.979693 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 08:32:56.839630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:56.839596 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:32:56.840208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:56.840135 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:32:56.841017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:56.840996 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:32:56.969517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:56.969484 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:32:56.970858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:32:56.970834 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:33:01.078379 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:01.078338 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh"] Nov 02 08:33:01.121380 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:01.121334 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f6fd3-2rrvh"] Nov 02 08:33:03.093935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:03.093896 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed9885e-9892-479b-838f-f3981a0271d7" path="/var/lib/kubelet/pods/6ed9885e-9892-479b-838f-f3981a0271d7/volumes" Nov 02 08:33:04.050359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:04.050326 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6"] Nov 02 08:33:04.096395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:04.096358 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5219343-rcln6"] Nov 02 08:33:05.104153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:05.104096 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2843f742-c60d-47ed-a8e8-e2553f858261" path="/var/lib/kubelet/pods/2843f742-c60d-47ed-a8e8-e2553f858261/volumes" Nov 02 08:33:05.733300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:05.733266 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv"] Nov 02 08:33:05.733504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:05.733470 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" containerID="cri-o://1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01" gracePeriod=30 Nov 02 08:33:11.047236 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.047206 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:33:11.163590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.163553 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.163801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.163742 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.163871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.163805 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.163871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.163858 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.163952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.163869 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:11.163952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.163929 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.164087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164057 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.164220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164102 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.164317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164266 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.164658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164416 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert\") pod \"12fb13e2-d141-4e0c-ab62-371193b96d49\" (UID: \"12fb13e2-d141-4e0c-ab62-371193b96d49\") " Nov 02 08:33:11.164658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164442 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:11.164658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164544 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:11.164658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164601 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:11.164909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.164826 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:11.165811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.165781 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.165940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.165815 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.165940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.165833 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.165940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.165847 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fb13e2-d141-4e0c-ab62-371193b96d49-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.165940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.165860 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.166665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.166637 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:33:11.166763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.166695 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:11.166885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.166858 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "kube-api-access-9jw5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:11.167068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.167050 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token" pod "12fb13e2-d141-4e0c-ab62-371193b96d49" (UID: "12fb13e2-d141-4e0c-ab62-371193b96d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:11.268180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.268074 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jw5g\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-kube-api-access-9jw5g\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.268180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.268109 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.268180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.268122 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.268180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.268134 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fb13e2-d141-4e0c-ab62-371193b96d49-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:11.455746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.455709 5055 generic.go:356] "Generic (PLEG): container finished" podID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerID="1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01" exitCode=0 Nov 02 08:33:11.455925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.455766 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" Nov 02 08:33:11.455925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.455800 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" event={"ID":"12fb13e2-d141-4e0c-ab62-371193b96d49","Type":"ContainerDied","Data":"1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01"} Nov 02 08:33:11.455925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.455869 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv" event={"ID":"12fb13e2-d141-4e0c-ab62-371193b96d49","Type":"ContainerDied","Data":"2a40fe6dd2c20215d2aa5fdb5ba67f8c28eefe4ee3e3ee8306093427bf2d1b21"} Nov 02 08:33:11.455925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.455898 5055 scope.go:122] "RemoveContainer" containerID="1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01" Nov 02 08:33:11.469057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.469026 5055 scope.go:122] "RemoveContainer" containerID="1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01" Nov 02 08:33:11.469425 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:33:11.469402 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01\": container with ID starting with 1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01 not found: ID does not exist" containerID="1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01" Nov 02 08:33:11.469500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.469439 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01"} err="failed to get container status \"1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01\": rpc error: code = NotFound desc = could not find container \"1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01\": container with ID starting with 1912379745a3cf7b0ab51d679121655a11b57f659fc4044cf1895b7dcbef4d01 not found: ID does not exist" Nov 02 08:33:11.614234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.614155 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv"] Nov 02 08:33:11.659879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:11.659846 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-lt5cv"] Nov 02 08:33:12.711979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.711941 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd"] Nov 02 08:33:12.712972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.712951 5055 cpu_manager.go:414] "containerMap: removing container" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" Nov 02 08:33:12.713035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.712976 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" Nov 02 08:33:12.713181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.713167 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" containerName="istio-proxy" Nov 02 08:33:12.717445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.717425 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.770035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.770006 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd"] Nov 02 08:33:12.810018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.809865 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:33:12.810018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.809933 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:33:12.810018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.809945 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:33:12.812032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.812006 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:33:12.896837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.896794 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.897043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.896932 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrtx\" (UniqueName: \"kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.897223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.897200 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.998916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.998808 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.999098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.998962 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.999098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.999062 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrtx\" (UniqueName: \"kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:12.999574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:12.999551 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:13.001561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:13.001537 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:13.057309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:13.057273 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrtx\" (UniqueName: \"kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx\") pod \"maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:13.094861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:13.094826 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12fb13e2-d141-4e0c-ab62-371193b96d49" path="/var/lib/kubelet/pods/12fb13e2-d141-4e0c-ab62-371193b96d49/volumes" Nov 02 08:33:13.332173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:13.332039 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:13.729299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:13.729256 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd"] Nov 02 08:33:13.730742 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:33:13.730711 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668c32ff_d359_43e6_b4f6_55bb4a307fff.slice/crio-6140d232ef57eef9480ada8a93a4cfb8deccf63c64705e10e32abe04ad57c6d7 WatchSource:0}: Error finding container 6140d232ef57eef9480ada8a93a4cfb8deccf63c64705e10e32abe04ad57c6d7: Status 404 returned error can't find the container with id 6140d232ef57eef9480ada8a93a4cfb8deccf63c64705e10e32abe04ad57c6d7 Nov 02 08:33:14.490337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:14.490299 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" event={"ID":"668c32ff-d359-43e6-b4f6-55bb4a307fff","Type":"ContainerStarted","Data":"c5287f556e2fac47805ad5ca6dd5c70552456bf9b69d7d4ce4790cccd160f9ed"} Nov 02 08:33:14.490538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:14.490357 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" event={"ID":"668c32ff-d359-43e6-b4f6-55bb4a307fff","Type":"ContainerStarted","Data":"6140d232ef57eef9480ada8a93a4cfb8deccf63c64705e10e32abe04ad57c6d7"} Nov 02 08:33:14.605849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:14.605771 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" podStartSLOduration=2.605752984 podStartE2EDuration="2.605752984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:14.60196587 +0000 UTC m=+1641.692586301" watchObservedRunningTime="2026-11-02 08:33:14.605752984 +0000 UTC m=+1641.696373404" Nov 02 08:33:14.652677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:14.652638 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm"] Nov 02 08:33:14.652885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:14.652861 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" containerID="cri-o://c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8" gracePeriod=30 Nov 02 08:33:15.504704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:15.504660 5055 generic.go:356] "Generic (PLEG): container finished" podID="668c32ff-d359-43e6-b4f6-55bb4a307fff" containerID="c5287f556e2fac47805ad5ca6dd5c70552456bf9b69d7d4ce4790cccd160f9ed" exitCode=0 Nov 02 08:33:15.505197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:15.504717 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" event={"ID":"668c32ff-d359-43e6-b4f6-55bb4a307fff","Type":"ContainerDied","Data":"c5287f556e2fac47805ad5ca6dd5c70552456bf9b69d7d4ce4790cccd160f9ed"} Nov 02 08:33:16.161275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.161237 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd"] Nov 02 08:33:16.209688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.209656 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.300162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.295862 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c17125-openshift-default-dockercfg-kkdmc\"" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374414 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374475 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374583 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374682 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374749 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374842 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374889 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.374949 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2pc\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.399179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.375018 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.472228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.468121 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd"] Nov 02 08:33:16.533173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.477309 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.542293 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.542261 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.542564 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.542544 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.542769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.542752 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.542897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.542884 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.543033 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.543020 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2pc\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.543202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.543177 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.543522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.543502 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.543640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.543627 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.544556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.544534 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.548859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.546808 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.548859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.547487 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.548859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.547495 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.548859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.547734 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.571783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.571713 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.572202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.572179 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.598847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.598799 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.600712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.600653 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2pc\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc\") pod \"e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.723005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.722937 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:16.746192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.746131 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:16.758548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.756313 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca\") pod \"668c32ff-d359-43e6-b4f6-55bb4a307fff\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " Nov 02 08:33:16.758548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.756433 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca\") pod \"668c32ff-d359-43e6-b4f6-55bb4a307fff\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " Nov 02 08:33:16.760272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.759249 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx\" (UniqueName: \"kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx\") pod \"668c32ff-d359-43e6-b4f6-55bb4a307fff\" (UID: \"668c32ff-d359-43e6-b4f6-55bb4a307fff\") " Nov 02 08:33:16.762715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.762679 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca" pod "668c32ff-d359-43e6-b4f6-55bb4a307fff" (UID: "668c32ff-d359-43e6-b4f6-55bb4a307fff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:16.765418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.765385 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx" pod "668c32ff-d359-43e6-b4f6-55bb4a307fff" (UID: "668c32ff-d359-43e6-b4f6-55bb4a307fff"). InnerVolumeSpecName "kube-api-access-zgrtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:16.765528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.765400 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca" pod "668c32ff-d359-43e6-b4f6-55bb4a307fff" (UID: "668c32ff-d359-43e6-b4f6-55bb4a307fff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:33:16.862575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.862482 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:16.862575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.862526 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgrtx\" (UniqueName: \"kubernetes.io/projected/668c32ff-d359-43e6-b4f6-55bb4a307fff-kube-api-access-zgrtx\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:16.862575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:16.862542 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668c32ff-d359-43e6-b4f6-55bb4a307fff-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:17.234535 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:33:17.234502 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe57046b_d6ac_417b_b856_2a940616bd25.slice/crio-377a56ebd577b5b51a3ff8f95300b1b5ea6261a27b0a8ff03b20be4a0a4a2ebb WatchSource:0}: Error finding container 377a56ebd577b5b51a3ff8f95300b1b5ea6261a27b0a8ff03b20be4a0a4a2ebb: Status 404 returned error can't find the container with id 377a56ebd577b5b51a3ff8f95300b1b5ea6261a27b0a8ff03b20be4a0a4a2ebb Nov 02 08:33:17.235082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.235056 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd"] Nov 02 08:33:17.237989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.237942 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:17.238090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.238037 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:17.238152 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.238094 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:17.545746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.545703 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" event={"ID":"be57046b-d6ac-417b-b856-2a940616bd25","Type":"ContainerStarted","Data":"e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1"} Nov 02 08:33:17.546273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.545762 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" event={"ID":"be57046b-d6ac-417b-b856-2a940616bd25","Type":"ContainerStarted","Data":"377a56ebd577b5b51a3ff8f95300b1b5ea6261a27b0a8ff03b20be4a0a4a2ebb"} Nov 02 08:33:17.547308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.547287 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" Nov 02 08:33:17.547308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.547296 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd" event={"ID":"668c32ff-d359-43e6-b4f6-55bb4a307fff","Type":"ContainerDied","Data":"6140d232ef57eef9480ada8a93a4cfb8deccf63c64705e10e32abe04ad57c6d7"} Nov 02 08:33:17.547468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.547341 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6140d232ef57eef9480ada8a93a4cfb8deccf63c64705e10e32abe04ad57c6d7" Nov 02 08:33:17.671870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.671812 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podStartSLOduration=1.6717973480000001 podStartE2EDuration="1.671797348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:17.667803804 +0000 UTC m=+1644.758424223" watchObservedRunningTime="2026-11-02 08:33:17.671797348 +0000 UTC m=+1644.762417766" Nov 02 08:33:17.748505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:17.748460 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:18.755260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:18.755227 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:19.564930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:19.564897 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:19.565974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:19.565953 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:33:19.977429 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:19.977400 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:33:20.016732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.016692 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.016927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.016822 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.016987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.016944 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.017037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017022 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.017096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017072 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:20.017737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017180 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:20.017737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017191 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.017737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017326 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:20.017737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017392 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.017737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.017394 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:20.018299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.018262 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.018413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.018302 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.018413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.018317 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.018413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.018330 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.019537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.019513 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:20.019693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.019676 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "kube-api-access-4rn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:20.119992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.119890 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.119992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.119957 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.119992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.119992 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy\") pod \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\" (UID: \"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5\") " Nov 02 08:33:20.120944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.120916 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.121032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.120949 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rn9k\" (UniqueName: \"kubernetes.io/projected/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-kube-api-access-4rn9k\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.121604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.121572 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:20.124245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.124219 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:33:20.127107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.127079 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy" pod "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" (UID: "510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:33:20.222761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.222723 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.222761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.222761 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.223003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.222775 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:20.573714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.573676 5055 generic.go:356] "Generic (PLEG): container finished" podID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerID="c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8" exitCode=0 Nov 02 08:33:20.573932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.573755 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" Nov 02 08:33:20.573932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.573765 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" event={"ID":"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5","Type":"ContainerDied","Data":"c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8"} Nov 02 08:33:20.573932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.573841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm" event={"ID":"510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5","Type":"ContainerDied","Data":"9b9af52bacb5751b4d2e1ef9b3017ac9ebc86c2947d23b306851b332e963187a"} Nov 02 08:33:20.573932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.573865 5055 scope.go:122] "RemoveContainer" containerID="c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8" Nov 02 08:33:20.590865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.590838 5055 scope.go:122] "RemoveContainer" containerID="c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8" Nov 02 08:33:20.591251 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:33:20.591227 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8\": container with ID starting with c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8 not found: ID does not exist" containerID="c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8" Nov 02 08:33:20.591347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.591262 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8"} err="failed to get container status \"c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8\": rpc error: code = NotFound desc = could not find container \"c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8\": container with ID starting with c2e9c2c9d0e2172a6ad6d8006c7a4791a010871db432ed3ef83387611ba0b1a8 not found: ID does not exist" Nov 02 08:33:20.733001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.732965 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm"] Nov 02 08:33:20.778505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:20.778463 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-7cpkm"] Nov 02 08:33:21.069241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:21.069200 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4"] Nov 02 08:33:21.073532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:21.073499 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835a782-tftc4"] Nov 02 08:33:21.094491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:21.094448 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" path="/var/lib/kubelet/pods/510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5/volumes" Nov 02 08:33:21.095081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:21.095057 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b14dd0-939b-4de2-9465-c655ac236d3f" path="/var/lib/kubelet/pods/68b14dd0-939b-4de2-9465-c655ac236d3f/volumes" Nov 02 08:33:24.571094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:24.570700 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:33:24.923554 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:33:24.923460 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:33:26.558170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.546045 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb"] Nov 02 08:33:26.560747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.560720 5055 cpu_manager.go:414] "containerMap: removing container" podUID="668c32ff-d359-43e6-b4f6-55bb4a307fff" containerName="revoke-keys" Nov 02 08:33:26.560925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.560912 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668c32ff-d359-43e6-b4f6-55bb4a307fff" containerName="revoke-keys" Nov 02 08:33:26.561070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.561060 5055 cpu_manager.go:414] "containerMap: removing container" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" Nov 02 08:33:26.568160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.561132 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" Nov 02 08:33:26.589167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.569445 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="510e13a2-4f9f-43f3-8fa7-9d8ddabce7e5" containerName="istio-proxy" Nov 02 08:33:26.589167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.569488 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="668c32ff-d359-43e6-b4f6-55bb4a307fff" containerName="revoke-keys" Nov 02 08:33:26.589167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.575815 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.626667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.626636 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:33:26.636475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.636441 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:33:26.653370 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.653333 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:33:26.715243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.712307 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.715243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.712425 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.715243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.712815 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrl5\" (UniqueName: \"kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.750077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.748331 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:33:26.751694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.751665 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb"] Nov 02 08:33:26.880964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.880857 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.897330 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.897297 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.898777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.898748 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrl5\" (UniqueName: \"kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.899379 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.898317 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.932228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.932194 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.954819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.954788 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrl5\" (UniqueName: \"kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5\") pod \"maas-api-revoke-sub-8ce12f9b-dlqbb\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:26.960670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:26.960642 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:27.366200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:27.366169 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb"] Nov 02 08:33:27.366476 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:33:27.366445 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e40792_3b31_49e5_8440_33e6250e99c6.slice/crio-a7fd06bb9835a489c7ad6e2c2c5cac68798750cc1f311487ec91cbd387f69854 WatchSource:0}: Error finding container a7fd06bb9835a489c7ad6e2c2c5cac68798750cc1f311487ec91cbd387f69854: Status 404 returned error can't find the container with id a7fd06bb9835a489c7ad6e2c2c5cac68798750cc1f311487ec91cbd387f69854 Nov 02 08:33:27.761664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:27.761621 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" event={"ID":"76e40792-3b31-49e5-8440-33e6250e99c6","Type":"ContainerStarted","Data":"a7fd06bb9835a489c7ad6e2c2c5cac68798750cc1f311487ec91cbd387f69854"} Nov 02 08:33:28.837423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:28.837184 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" event={"ID":"76e40792-3b31-49e5-8440-33e6250e99c6","Type":"ContainerStarted","Data":"ee7b9224e53861c4d3135128ee6729036ecb537bf0974898c06bacc082193437"} Nov 02 08:33:28.951902 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:28.951866 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm"] Nov 02 08:33:28.957001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:28.956975 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.010223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.009464 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b6a289-openshift-default-dockercfg-x79sn\"" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011087 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011175 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011211 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011269 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011366 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011431 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldf7\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011511 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011641 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.040316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.011660 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.065979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.064656 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm"] Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141214 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141380 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141483 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141553 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldf7\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141636 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141771 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.141802 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.144418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.142328 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.159904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.152425 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.159904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.154995 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.159904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.155553 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.159904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.156823 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.161194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.160969 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.554164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.533458 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldf7\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.554164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.538393 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.554164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.538604 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy\") pod \"e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.619166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.606478 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:29.849891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.849794 5055 generic.go:356] "Generic (PLEG): container finished" podID="76e40792-3b31-49e5-8440-33e6250e99c6" containerID="ee7b9224e53861c4d3135128ee6729036ecb537bf0974898c06bacc082193437" exitCode=0 Nov 02 08:33:29.850358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:29.849887 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" event={"ID":"76e40792-3b31-49e5-8440-33e6250e99c6","Type":"ContainerDied","Data":"ee7b9224e53861c4d3135128ee6729036ecb537bf0974898c06bacc082193437"} Nov 02 08:33:30.040192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.040161 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:30.113484 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:33:30.113444 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048aa870_d17c_47ee_9ed0_583b3324f9d8.slice/crio-d7e2fc15669131b09dc6052c09e313449c1305b97f890895ef5493d343feb770 WatchSource:0}: Error finding container d7e2fc15669131b09dc6052c09e313449c1305b97f890895ef5493d343feb770: Status 404 returned error can't find the container with id d7e2fc15669131b09dc6052c09e313449c1305b97f890895ef5493d343feb770 Nov 02 08:33:30.116505 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.116466 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:30.116606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.116545 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:30.116606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.116576 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:30.155169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.154899 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm"] Nov 02 08:33:30.236807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.236781 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca\") pod \"76e40792-3b31-49e5-8440-33e6250e99c6\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " Nov 02 08:33:30.236942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.236850 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5\" (UniqueName: \"kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5\") pod \"76e40792-3b31-49e5-8440-33e6250e99c6\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " Nov 02 08:33:30.237272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.237169 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca\") pod \"76e40792-3b31-49e5-8440-33e6250e99c6\" (UID: \"76e40792-3b31-49e5-8440-33e6250e99c6\") " Nov 02 08:33:30.237819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.237511 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca" pod "76e40792-3b31-49e5-8440-33e6250e99c6" (UID: "76e40792-3b31-49e5-8440-33e6250e99c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:30.237819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.237790 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:30.241864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.241837 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5" pod "76e40792-3b31-49e5-8440-33e6250e99c6" (UID: "76e40792-3b31-49e5-8440-33e6250e99c6"). InnerVolumeSpecName "kube-api-access-7wrl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:30.242266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.242244 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca" pod "76e40792-3b31-49e5-8440-33e6250e99c6" (UID: "76e40792-3b31-49e5-8440-33e6250e99c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:33:30.349167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.348854 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e40792-3b31-49e5-8440-33e6250e99c6-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:30.349167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.348890 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrl5\" (UniqueName: \"kubernetes.io/projected/76e40792-3b31-49e5-8440-33e6250e99c6-kube-api-access-7wrl5\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:30.860512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.860471 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" event={"ID":"048aa870-d17c-47ee-9ed0-583b3324f9d8","Type":"ContainerStarted","Data":"5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b"} Nov 02 08:33:30.861020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.860531 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" event={"ID":"048aa870-d17c-47ee-9ed0-583b3324f9d8","Type":"ContainerStarted","Data":"d7e2fc15669131b09dc6052c09e313449c1305b97f890895ef5493d343feb770"} Nov 02 08:33:30.862056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.862022 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" event={"ID":"76e40792-3b31-49e5-8440-33e6250e99c6","Type":"ContainerDied","Data":"a7fd06bb9835a489c7ad6e2c2c5cac68798750cc1f311487ec91cbd387f69854"} Nov 02 08:33:30.862056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.862040 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb" Nov 02 08:33:30.862259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:30.862069 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fd06bb9835a489c7ad6e2c2c5cac68798750cc1f311487ec91cbd387f69854" Nov 02 08:33:31.028399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:31.028340 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podStartSLOduration=3.028325601 podStartE2EDuration="3.028325601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:31.024829541 +0000 UTC m=+1658.115449974" watchObservedRunningTime="2026-11-02 08:33:31.028325601 +0000 UTC m=+1658.118946020" Nov 02 08:33:31.617226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:31.609561 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:31.619522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:31.619489 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:31.870187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:31.870077 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:31.870958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:31.870937 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:33:36.895647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.895594 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf"] Nov 02 08:33:36.896573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.896552 5055 cpu_manager.go:414] "containerMap: removing container" podUID="76e40792-3b31-49e5-8440-33e6250e99c6" containerName="revoke-keys" Nov 02 08:33:36.896573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.896573 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e40792-3b31-49e5-8440-33e6250e99c6" containerName="revoke-keys" Nov 02 08:33:36.896740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.896729 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e40792-3b31-49e5-8440-33e6250e99c6" containerName="revoke-keys" Nov 02 08:33:36.899451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.899428 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.926847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.926814 5055 generic.go:356] "Generic (PLEG): container finished" podID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerID="389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907" exitCode=0 Nov 02 08:33:36.927049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.926883 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" event={"ID":"d475b9a6-835f-4ac5-a681-fb2346b2976e","Type":"ContainerDied","Data":"389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907"} Nov 02 08:33:36.927407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.927390 5055 scope.go:122] "RemoveContainer" containerID="389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907" Nov 02 08:33:36.929368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.929332 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:36.929483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.929425 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:36.929553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.929478 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:36.990230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990199 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6cq\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990421 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990285 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990434 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990486 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990511 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990547 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990602 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990654 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:36.990835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:36.990817 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.092159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.092096 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.092417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.092393 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.092722 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.092707 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.092874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.092860 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6cq\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.093063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093050 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.093279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093251 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.093471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093455 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.093610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.093724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093712 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.093871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093849 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.094489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.094463 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.094615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.093521 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.095334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.095084 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.095639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.095590 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.098597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.098501 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.103864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.102113 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf"] Nov 02 08:33:37.103864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.103773 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.150215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.150113 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.154443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.154416 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6cq\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.219430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.219394 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:37.694338 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:33:37.694292 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf55650_e5c8_49d1_b9ab_fcd5887c905b.slice/crio-abd4cf1d486d72a4c9fa172900437a9173042ae3a2e9d381701dc5b85ac7dfd3 WatchSource:0}: Error finding container abd4cf1d486d72a4c9fa172900437a9173042ae3a2e9d381701dc5b85ac7dfd3: Status 404 returned error can't find the container with id abd4cf1d486d72a4c9fa172900437a9173042ae3a2e9d381701dc5b85ac7dfd3 Nov 02 08:33:37.694621 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.694595 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf"] Nov 02 08:33:37.697924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.697883 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:37.698025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.697970 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:37.698025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.698015 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:33:37.957417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.957355 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" event={"ID":"d475b9a6-835f-4ac5-a681-fb2346b2976e","Type":"ContainerStarted","Data":"f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b"} Nov 02 08:33:37.961895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.961864 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" event={"ID":"fcf55650-e5c8-49d1-b9ab-fcd5887c905b","Type":"ContainerStarted","Data":"7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489"} Nov 02 08:33:37.962044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:37.961912 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" event={"ID":"fcf55650-e5c8-49d1-b9ab-fcd5887c905b","Type":"ContainerStarted","Data":"abd4cf1d486d72a4c9fa172900437a9173042ae3a2e9d381701dc5b85ac7dfd3"} Nov 02 08:33:38.008281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.008240 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:33:38.217212 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.213056 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podStartSLOduration=2.213034207 podStartE2EDuration="2.213034207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:38.209679838 +0000 UTC m=+1665.300300271" watchObservedRunningTime="2026-11-02 08:33:38.213034207 +0000 UTC m=+1665.303654627" Nov 02 08:33:38.227165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.225599 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:38.227165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.226076 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:38.227165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.226160 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:38.340167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.340121 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:33:38.342763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.342727 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 08:33:38.342920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:38.342811 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 08:33:39.274169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:39.249271 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:39.274169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:39.249358 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:39.430217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:39.399259 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 08:33:39.430217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:39.399338 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 08:33:40.240162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:40.240042 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:40.240162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:40.240118 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:40.248291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:40.248240 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 08:33:40.349843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:40.349271 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 08:33:40.349843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:40.349343 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 08:33:41.236231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:41.235691 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:41.236231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:41.235752 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:41.346164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:41.342530 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 08:33:41.346164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:41.342591 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 08:33:42.234166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:42.228515 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:42.234166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:42.228580 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:42.345794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:42.345378 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 08:33:42.345794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:42.345426 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 08:33:43.225163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:43.222670 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:43.225163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:43.222744 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:43.347319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:43.347268 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 08:33:43.347783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:43.347357 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 08:33:44.220620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:44.220581 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:44.220985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:44.220949 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:44.369227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:44.369192 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:33:44.371677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:44.371644 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:33:45.061590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:45.061552 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5"] Nov 02 08:33:45.070979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:45.070951 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ebfca5-ghvd5"] Nov 02 08:33:45.123494 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:45.123459 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08548a78-25ad-4f26-89af-cd304ca47a67" path="/var/lib/kubelet/pods/08548a78-25ad-4f26-89af-cd304ca47a67/volumes" Nov 02 08:33:45.219777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:45.219741 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 08:33:45.219975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:45.219812 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 08:33:46.238530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:46.238491 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:46.239462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:46.239431 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:46.242376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:46.242337 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:33:53.051000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:53.050961 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7"] Nov 02 08:33:53.096880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:53.096847 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7"] Nov 02 08:33:53.138031 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:33:53.137990 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-6eb3548f-kzpz7\" is forbidden: User \"system:node:ip-10-0-139-247.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-247.us-west-2.compute.internal' and this object" podUID="13335172-1491-4c6b-b75c-58fbc39bf141" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb3548f-kzpz7" Nov 02 08:33:54.163555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:54.163523 5055 scope.go:122] "RemoveContainer" containerID="43502cea273e146487c955ff00a34901ea388e18db8eff6f6839c1f4fec5162f" Nov 02 08:33:54.174595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:54.174566 5055 scope.go:122] "RemoveContainer" containerID="f5b367eda82e5cc62465d1422346ad2b195111e88f23eadd44ee7c4636ee5aa2" Nov 02 08:33:54.185051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:54.185023 5055 scope.go:122] "RemoveContainer" containerID="31613fa51c949668e9866e6771ed2d2f3be8f5322c714275628ec743a250b17d" Nov 02 08:33:54.195115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:54.195089 5055 scope.go:122] "RemoveContainer" containerID="8dbd01d34fcfb1c33ce7e610501deff635976bd93111d842b0b2422a0e27bac1" Nov 02 08:33:54.205479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:54.205453 5055 scope.go:122] "RemoveContainer" containerID="da272c0ce48ea2eb489b040a4d0e1b8cc3a1e35662cbaa02c34c716c4d063672" Nov 02 08:33:54.944310 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:33:54.944270 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:33:55.094489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:55.094452 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13335172-1491-4c6b-b75c-58fbc39bf141" path="/var/lib/kubelet/pods/13335172-1491-4c6b-b75c-58fbc39bf141/volumes" Nov 02 08:33:57.051396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:57.051356 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b"] Nov 02 08:33:57.096713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:57.096671 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cbb200-xnf6b"] Nov 02 08:33:58.409235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.409193 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c"] Nov 02 08:33:58.416243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.416207 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.469573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.469522 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c"] Nov 02 08:33:58.483191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.483127 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkk85\" (UniqueName: \"kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.483369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.483293 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.483425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.483404 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.509187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.509104 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:33:58.509382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.509181 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:33:58.510708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.510681 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:33:58.510775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.510713 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:33:58.585104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.585051 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkk85\" (UniqueName: \"kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.585366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.585127 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.585366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.585225 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.586130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.586095 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.587820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.587794 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.638775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.638732 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkk85\" (UniqueName: \"kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85\") pod \"maas-api-revoke-keys-e2e-api-a-b26f03-blp7c\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:58.734410 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:58.734376 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:33:59.096054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:59.095965 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9de232-a7ce-43a0-a1a6-add363636639" path="/var/lib/kubelet/pods/dc9de232-a7ce-43a0-a1a6-add363636639/volumes" Nov 02 08:33:59.114819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:59.114786 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c"] Nov 02 08:33:59.115779 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:33:59.115753 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb781d3_f800_4d77_b187_4dd63e51f331.slice/crio-691d46afe0c3cf991eb476230b9777706f5f4d3bea7c3b3acabe7cbf8949e0d4 WatchSource:0}: Error finding container 691d46afe0c3cf991eb476230b9777706f5f4d3bea7c3b3acabe7cbf8949e0d4: Status 404 returned error can't find the container with id 691d46afe0c3cf991eb476230b9777706f5f4d3bea7c3b3acabe7cbf8949e0d4 Nov 02 08:33:59.370681 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:33:59.370581 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" event={"ID":"5eb781d3-f800-4d77-b187-4dd63e51f331","Type":"ContainerStarted","Data":"691d46afe0c3cf991eb476230b9777706f5f4d3bea7c3b3acabe7cbf8949e0d4"} Nov 02 08:34:00.382825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:00.382780 5055 generic.go:356] "Generic (PLEG): container finished" podID="5eb781d3-f800-4d77-b187-4dd63e51f331" containerID="be0a089d14f0acda65bae694d1b6abd2fcc764b10391403552295fa60049fcdf" exitCode=0 Nov 02 08:34:00.383337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:00.382873 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" event={"ID":"5eb781d3-f800-4d77-b187-4dd63e51f331","Type":"ContainerDied","Data":"be0a089d14f0acda65bae694d1b6abd2fcc764b10391403552295fa60049fcdf"} Nov 02 08:34:01.583867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.583835 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:34:01.734493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.734458 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85\" (UniqueName: \"kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85\") pod \"5eb781d3-f800-4d77-b187-4dd63e51f331\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " Nov 02 08:34:01.734691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.734513 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca\") pod \"5eb781d3-f800-4d77-b187-4dd63e51f331\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " Nov 02 08:34:01.734691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.734631 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca\") pod \"5eb781d3-f800-4d77-b187-4dd63e51f331\" (UID: \"5eb781d3-f800-4d77-b187-4dd63e51f331\") " Nov 02 08:34:01.734979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.734948 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca" pod "5eb781d3-f800-4d77-b187-4dd63e51f331" (UID: "5eb781d3-f800-4d77-b187-4dd63e51f331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:01.735436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.735411 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:01.736932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.736897 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85" pod "5eb781d3-f800-4d77-b187-4dd63e51f331" (UID: "5eb781d3-f800-4d77-b187-4dd63e51f331"). InnerVolumeSpecName "kube-api-access-xkk85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:01.737299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.737275 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca" pod "5eb781d3-f800-4d77-b187-4dd63e51f331" (UID: "5eb781d3-f800-4d77-b187-4dd63e51f331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:34:01.836959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.836909 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb781d3-f800-4d77-b187-4dd63e51f331-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:01.836959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:01.836953 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkk85\" (UniqueName: \"kubernetes.io/projected/5eb781d3-f800-4d77-b187-4dd63e51f331-kube-api-access-xkk85\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:02.401277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:02.401244 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" Nov 02 08:34:02.401277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:02.401259 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c" event={"ID":"5eb781d3-f800-4d77-b187-4dd63e51f331","Type":"ContainerDied","Data":"691d46afe0c3cf991eb476230b9777706f5f4d3bea7c3b3acabe7cbf8949e0d4"} Nov 02 08:34:02.401515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:02.401307 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="691d46afe0c3cf991eb476230b9777706f5f4d3bea7c3b3acabe7cbf8949e0d4" Nov 02 08:34:09.534860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:09.534809 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:34:19.053620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:19.053581 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb"] Nov 02 08:34:19.099247 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:19.099214 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838627ed-t64nb"] Nov 02 08:34:21.095188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:21.095135 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae80f6a0-7bcd-489f-a6e8-7eb0146757cd" path="/var/lib/kubelet/pods/ae80f6a0-7bcd-489f-a6e8-7eb0146757cd/volumes" Nov 02 08:34:22.224925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.224892 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz"] Nov 02 08:34:22.225606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.225585 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5eb781d3-f800-4d77-b187-4dd63e51f331" containerName="revoke-keys" Nov 02 08:34:22.225606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.225607 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb781d3-f800-4d77-b187-4dd63e51f331" containerName="revoke-keys" Nov 02 08:34:22.225759 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.225734 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb781d3-f800-4d77-b187-4dd63e51f331" containerName="revoke-keys" Nov 02 08:34:22.229209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.229191 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.330708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.330672 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz"] Nov 02 08:34:22.357374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357297 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357351 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357409 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357504 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357541 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357601 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357813 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kx4\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.357986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.357845 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.358117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.358034 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.458973 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459150 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459185 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459242 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459336 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459369 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459603 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459432 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459603 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459499 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459603 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459536 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kx4\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459603 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459562 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.459823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.459774 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.460108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.460083 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.460333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.460312 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.460429 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.460407 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.462317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.462295 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.462830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.462805 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.513312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.513237 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.513464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.513321 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kx4\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.544560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.544523 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:22.972551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:22.972518 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz"] Nov 02 08:34:23.008114 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:34:23.008083 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1877c01_18c5_489c_8847_d4c15e388b02.slice/crio-b015db53ee881b16a5499c4e6cfa364c66f5d4908c3f353e00a71b276b11047a WatchSource:0}: Error finding container b015db53ee881b16a5499c4e6cfa364c66f5d4908c3f353e00a71b276b11047a: Status 404 returned error can't find the container with id b015db53ee881b16a5499c4e6cfa364c66f5d4908c3f353e00a71b276b11047a Nov 02 08:34:23.010918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:23.010881 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:34:23.010994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:23.010957 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:34:23.010994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:23.010984 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:34:23.852598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:23.852556 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" event={"ID":"b1877c01-18c5-489c-8847-d4c15e388b02","Type":"ContainerStarted","Data":"c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc"} Nov 02 08:34:23.853090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:23.852650 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" event={"ID":"b1877c01-18c5-489c-8847-d4c15e388b02","Type":"ContainerStarted","Data":"b015db53ee881b16a5499c4e6cfa364c66f5d4908c3f353e00a71b276b11047a"} Nov 02 08:34:24.013318 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:24.013243 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podStartSLOduration=2.013223129 podStartE2EDuration="2.013223129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:24.010728891 +0000 UTC m=+1711.101349312" watchObservedRunningTime="2026-11-02 08:34:24.013223129 +0000 UTC m=+1711.103843546" Nov 02 08:34:24.545728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:24.545686 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:24.547725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:24.547699 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 08:34:24.547812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:24.547755 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 08:34:24.934577 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:34:24.934477 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:34:25.545024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:25.544973 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 08:34:25.545297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:25.545050 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 08:34:26.544924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:26.544879 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 08:34:26.545427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:26.544953 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 08:34:27.545260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:27.545219 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 08:34:27.545712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:27.545305 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 08:34:28.545877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:28.545836 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 08:34:28.546504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:28.545912 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 08:34:29.550443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:29.550411 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:29.550905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:29.550882 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:29.551811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:29.551790 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:34:30.055697 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:30.050212 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9"] Nov 02 08:34:30.058474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:30.058438 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224327da-msqz9"] Nov 02 08:34:31.094594 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:31.094555 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595f472a-4b68-408d-b382-71d104b1f0c5" path="/var/lib/kubelet/pods/595f472a-4b68-408d-b382-71d104b1f0c5/volumes" Nov 02 08:34:36.130252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:36.130197 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2"] Nov 02 08:34:36.145936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:36.145880 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd78a964-g6lf2"] Nov 02 08:34:37.108213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:37.108170 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a2fa93-b7dd-4d19-a806-191475ea3e4c" path="/var/lib/kubelet/pods/b1a2fa93-b7dd-4d19-a806-191475ea3e4c/volumes" Nov 02 08:34:39.415305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:39.415258 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll"] Nov 02 08:34:39.415810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:39.415545 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" containerID="cri-o://316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e" gracePeriod=30 Nov 02 08:34:44.762905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.762874 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:34:44.808320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808283 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.808525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808394 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.808525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808436 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.808525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808487 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.808704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808528 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.808802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808747 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.808867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808814 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:44.808979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808954 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.809069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808985 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:44.809069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.809009 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.809214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.809073 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l\") pod \"c800c0aa-716d-40aa-8709-70274d48d9d8\" (UID: \"c800c0aa-716d-40aa-8709-70274d48d9d8\") " Nov 02 08:34:44.809274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.809253 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:44.809337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.808981 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:44.809520 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.809495 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:44.810285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.810257 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.810411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.810287 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c800c0aa-716d-40aa-8709-70274d48d9d8-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.810411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.810304 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.810411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.810318 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.810411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.810335 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.810901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.810877 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:44.811117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.811093 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:34:44.811193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.811180 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:44.811402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.811382 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l" pod "c800c0aa-716d-40aa-8709-70274d48d9d8" (UID: "c800c0aa-716d-40aa-8709-70274d48d9d8"). InnerVolumeSpecName "kube-api-access-b5c6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:44.911787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.911742 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.911787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.911783 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.911787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.911796 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c800c0aa-716d-40aa-8709-70274d48d9d8-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:44.912030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:44.911809 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5c6l\" (UniqueName: \"kubernetes.io/projected/c800c0aa-716d-40aa-8709-70274d48d9d8-kube-api-access-b5c6l\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:45.321027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.320990 5055 generic.go:356] "Generic (PLEG): container finished" podID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerID="316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e" exitCode=0 Nov 02 08:34:45.321268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.321044 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" Nov 02 08:34:45.321268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.321048 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" event={"ID":"c800c0aa-716d-40aa-8709-70274d48d9d8","Type":"ContainerDied","Data":"316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e"} Nov 02 08:34:45.321268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.321193 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll" event={"ID":"c800c0aa-716d-40aa-8709-70274d48d9d8","Type":"ContainerDied","Data":"a57d65646d4aad83c490c94e7182382469e0256bf6c22c3def601cb6d34032a4"} Nov 02 08:34:45.321268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.321220 5055 scope.go:122] "RemoveContainer" containerID="316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e" Nov 02 08:34:45.350204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.350174 5055 scope.go:122] "RemoveContainer" containerID="316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e" Nov 02 08:34:45.350662 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:34:45.350625 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e\": container with ID starting with 316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e not found: ID does not exist" containerID="316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e" Nov 02 08:34:45.350726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.350684 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e"} err="failed to get container status \"316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e\": rpc error: code = NotFound desc = could not find container \"316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e\": container with ID starting with 316c42d5d58b8f822b75421cffca90715f11db8473a0f8067421a7cc41dbfc7e not found: ID does not exist" Nov 02 08:34:45.476045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.476009 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll"] Nov 02 08:34:45.498163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:45.496249 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-qj7ll"] Nov 02 08:34:46.861011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:46.860974 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8"] Nov 02 08:34:46.861500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:46.861277 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" containerID="cri-o://9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9" gracePeriod=30 Nov 02 08:34:46.909696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:46.909649 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg"] Nov 02 08:34:46.909936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:46.909911 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" containerID="cri-o://ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181" gracePeriod=30 Nov 02 08:34:46.916221 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:46.916186 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf"] Nov 02 08:34:46.916482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:46.916454 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" containerID="cri-o://5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5" gracePeriod=30 Nov 02 08:34:47.046331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:47.046291 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z"] Nov 02 08:34:47.052234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:47.052193 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4de34-zxx7z"] Nov 02 08:34:47.096417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:47.096376 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf9096f-fa91-4819-a24a-a7eb15ec5ba3" path="/var/lib/kubelet/pods/bcf9096f-fa91-4819-a24a-a7eb15ec5ba3/volumes" Nov 02 08:34:47.097032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:47.097012 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" path="/var/lib/kubelet/pods/c800c0aa-716d-40aa-8709-70274d48d9d8/volumes" Nov 02 08:34:52.267667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.267631 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:34:52.296766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.296730 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:34:52.300852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.300820 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:34:52.392739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.392703 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.392927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.392765 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.392927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.392810 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.392927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.392866 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.392927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.392895 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.393535 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.393494 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.393795 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.393841 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.393934 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394078 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394106 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394129 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394207 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394251 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394320 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394402 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394469 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394502 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.394559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394535 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394582 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394624 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394661 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394719 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394775 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394858 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394894 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket\") pod \"801546db-f697-46a3-a7df-cc99ae52131b\" (UID: \"801546db-f697-46a3-a7df-cc99ae52131b\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394954 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.394984 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token\") pod \"b4dca4a6-8732-4d46-970c-f5db40b95df9\" (UID: \"b4dca4a6-8732-4d46-970c-f5db40b95df9\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.395128 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.395333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.395223 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data\") pod \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\" (UID: \"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd\") " Nov 02 08:34:52.395924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.395713 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.395924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.395892 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.396070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.393753 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.396070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396056 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.396290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396198 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396220 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396235 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396253 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396269 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396283 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396296 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.396590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396338 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.396590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396503 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.396590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396515 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:52.396823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396789 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:52.397203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.396868 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:52.397203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.397071 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.397426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.397310 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.397504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.397492 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.397573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.397519 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:34:52.397645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.397583 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.397645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.397595 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "kube-api-access-hcs85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:52.399506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.399481 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:52.400946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.400923 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w" pod "801546db-f697-46a3-a7df-cc99ae52131b" (UID: "801546db-f697-46a3-a7df-cc99ae52131b"). InnerVolumeSpecName "kube-api-access-fj96w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:52.401131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.401105 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:34:52.401287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.401264 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:52.401428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.401404 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "kube-api-access-kmth9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:52.401532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.401437 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:52.401532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.401457 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:34:52.402441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.402416 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy" pod "b4dca4a6-8732-4d46-970c-f5db40b95df9" (UID: "b4dca4a6-8732-4d46-970c-f5db40b95df9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.402693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.402670 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy" pod "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" (UID: "3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:52.402894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.402869 5055 generic.go:356] "Generic (PLEG): container finished" podID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerID="ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181" exitCode=0 Nov 02 08:34:52.402985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.402965 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" Nov 02 08:34:52.403081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.402992 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" event={"ID":"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd","Type":"ContainerDied","Data":"ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181"} Nov 02 08:34:52.403081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.403061 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg" event={"ID":"3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd","Type":"ContainerDied","Data":"f5275781982ec321509705e49635898c6ac3c73a404ac5df5546f2d0875e6c7e"} Nov 02 08:34:52.403220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.403094 5055 scope.go:122] "RemoveContainer" containerID="ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181" Nov 02 08:34:52.404939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.404917 5055 generic.go:356] "Generic (PLEG): container finished" podID="801546db-f697-46a3-a7df-cc99ae52131b" containerID="5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5" exitCode=0 Nov 02 08:34:52.405026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.404964 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" Nov 02 08:34:52.405026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.404996 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" event={"ID":"801546db-f697-46a3-a7df-cc99ae52131b","Type":"ContainerDied","Data":"5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5"} Nov 02 08:34:52.405310 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.405042 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf" event={"ID":"801546db-f697-46a3-a7df-cc99ae52131b","Type":"ContainerDied","Data":"12f73d516096a1c6782d8b5b34ac6f9d6fa8ee61cf3cfaafc14b130f2b4eaa7a"} Nov 02 08:34:52.406854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.406743 5055 generic.go:356] "Generic (PLEG): container finished" podID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerID="9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9" exitCode=0 Nov 02 08:34:52.406854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.406795 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" Nov 02 08:34:52.406854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.406821 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" event={"ID":"b4dca4a6-8732-4d46-970c-f5db40b95df9","Type":"ContainerDied","Data":"9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9"} Nov 02 08:34:52.407072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.406864 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8" event={"ID":"b4dca4a6-8732-4d46-970c-f5db40b95df9","Type":"ContainerDied","Data":"336e01b4ec32c65ac5a2d82d493a881fc5ccf07a120e3fa962847a4c91d31744"} Nov 02 08:34:52.418167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.418040 5055 scope.go:122] "RemoveContainer" containerID="ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181" Nov 02 08:34:52.418635 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:34:52.418608 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181\": container with ID starting with ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181 not found: ID does not exist" containerID="ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181" Nov 02 08:34:52.418796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.418646 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181"} err="failed to get container status \"ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181\": rpc error: code = NotFound desc = could not find container \"ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181\": container with ID starting with ea0acfed01086c29e633ba290a63cfdf31971b26fc5c2ed2b4c1a47b89287181 not found: ID does not exist" Nov 02 08:34:52.418796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.418670 5055 scope.go:122] "RemoveContainer" containerID="5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5" Nov 02 08:34:52.432791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.432769 5055 scope.go:122] "RemoveContainer" containerID="5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5" Nov 02 08:34:52.433135 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:34:52.433107 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5\": container with ID starting with 5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5 not found: ID does not exist" containerID="5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5" Nov 02 08:34:52.433230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.433158 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5"} err="failed to get container status \"5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5\": rpc error: code = NotFound desc = could not find container \"5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5\": container with ID starting with 5a61cdb2c145ae42183e3538183c49b267b933055e3fe307ef9f7448ae37a2b5 not found: ID does not exist" Nov 02 08:34:52.433230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.433182 5055 scope.go:122] "RemoveContainer" containerID="9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9" Nov 02 08:34:52.450181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.450157 5055 scope.go:122] "RemoveContainer" containerID="9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9" Nov 02 08:34:52.450546 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:34:52.450523 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9\": container with ID starting with 9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9 not found: ID does not exist" containerID="9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9" Nov 02 08:34:52.450619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.450553 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9"} err="failed to get container status \"9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9\": rpc error: code = NotFound desc = could not find container \"9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9\": container with ID starting with 9313145bb8d880b2a7d9973e3f9b337b61f3bede253dbb4e03802da5ec974ce9 not found: ID does not exist" Nov 02 08:34:52.497392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497351 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497380 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497394 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmth9\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-kube-api-access-kmth9\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497404 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497413 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497422 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801546db-f697-46a3-a7df-cc99ae52131b-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497431 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dca4a6-8732-4d46-970c-f5db40b95df9-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497443 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497456 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497465 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497473 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497482 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcs85\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-kube-api-access-hcs85\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497493 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801546db-f697-46a3-a7df-cc99ae52131b-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497502 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497510 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497519 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dca4a6-8732-4d46-970c-f5db40b95df9-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497533 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj96w\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-kube-api-access-fj96w\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497544 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801546db-f697-46a3-a7df-cc99ae52131b-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497553 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.497699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.497561 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801546db-f697-46a3-a7df-cc99ae52131b-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:52.611334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.611287 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf"] Nov 02 08:34:52.615269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.615243 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-4vhlf"] Nov 02 08:34:52.708610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.708573 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg"] Nov 02 08:34:52.712802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.712771 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-cbfzg"] Nov 02 08:34:52.875944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.875907 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8"] Nov 02 08:34:52.879758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:52.879724 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-phrb8"] Nov 02 08:34:53.125821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:53.125736 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" path="/var/lib/kubelet/pods/3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd/volumes" Nov 02 08:34:53.126908 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:53.126875 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801546db-f697-46a3-a7df-cc99ae52131b" path="/var/lib/kubelet/pods/801546db-f697-46a3-a7df-cc99ae52131b/volumes" Nov 02 08:34:53.127592 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:53.127571 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" path="/var/lib/kubelet/pods/b4dca4a6-8732-4d46-970c-f5db40b95df9/volumes" Nov 02 08:34:54.268298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:54.268250 5055 scope.go:122] "RemoveContainer" containerID="d5151fdefb22ec6e3b58be20cf86fd06dceb80d63647db57bc0d0c84a0ad6e36" Nov 02 08:34:54.307736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:54.307227 5055 scope.go:122] "RemoveContainer" containerID="c1982f5d36507c40edeb01ba3c848a38358879f650f6a9a19edb9e72322c21fb" Nov 02 08:34:54.326204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:54.326135 5055 scope.go:122] "RemoveContainer" containerID="559ab60be65f98d50b78323dc681954b79d0b2b0e8352fc748fc35ea575a19dd" Nov 02 08:34:54.352955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:54.352911 5055 scope.go:122] "RemoveContainer" containerID="8a50c28495bec7a014c7d409070bb38b8dead75f851be0b5667bf4cf28b31b96" Nov 02 08:34:54.372299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:54.372271 5055 scope.go:122] "RemoveContainer" containerID="805f9d6a7d25d8084b786e6476cfd5a049ee7bcd758d1df37f22e3e03dea405c" Nov 02 08:34:54.399312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:54.399276 5055 scope.go:122] "RemoveContainer" containerID="370b0a815b1765e5847ec558ce2e04f64236f1e7a45313540e4ff9e68d4411df" Nov 02 08:34:54.928299 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:34:54.928249 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:34:55.888875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:55.888835 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b"] Nov 02 08:34:55.932417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:55.932374 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-szr7b"] Nov 02 08:34:57.095116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:34:57.095083 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fa51d4-6d34-45c9-877d-0a95054f1331" path="/var/lib/kubelet/pods/58fa51d4-6d34-45c9-877d-0a95054f1331/volumes" Nov 02 08:35:19.981999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:19.981965 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:35:24.935375 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:35:24.935337 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:35:31.864004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:31.863967 5055 generic.go:356] "Generic (PLEG): container finished" podID="be57046b-d6ac-417b-b856-2a940616bd25" containerID="e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1" exitCode=0 Nov 02 08:35:31.864497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:31.864041 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" event={"ID":"be57046b-d6ac-417b-b856-2a940616bd25","Type":"ContainerDied","Data":"e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1"} Nov 02 08:35:31.864576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:31.864561 5055 scope.go:122] "RemoveContainer" containerID="e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1" Nov 02 08:35:31.866378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:31.866347 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:35:31.866523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:31.866418 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:35:31.866523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:31.866448 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:35:32.297647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:32.297608 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd"] Nov 02 08:35:32.301155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:32.301119 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5bb4f-8n5zd"] Nov 02 08:35:32.875687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:32.875648 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" event={"ID":"be57046b-d6ac-417b-b856-2a940616bd25","Type":"ContainerStarted","Data":"dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc"} Nov 02 08:35:33.116270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:33.116234 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668c32ff-d359-43e6-b4f6-55bb4a307fff" path="/var/lib/kubelet/pods/668c32ff-d359-43e6-b4f6-55bb4a307fff/volumes" Nov 02 08:35:33.748425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:33.748379 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:35:33.749792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:33.749753 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:35:33.749947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:33.749808 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 08:35:34.565438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:34.565397 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:35:34.749175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:34.749129 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:35:34.749433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:34.749401 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 08:35:35.748903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:35.748872 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:35:35.749437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:35.749058 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 08:35:36.748569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:36.748534 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:35:36.748788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:36.748606 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 08:35:37.749643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:37.749597 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:35:37.750171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:37.749659 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 08:35:38.768311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:38.768266 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:35:38.774287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:38.774115 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:35:38.976935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:38.976895 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s"] Nov 02 08:35:38.977724 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:38.977441 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" containerID="cri-o://838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7" gracePeriod=30 Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.321931 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc"] Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.322930 5055 cpu_manager.go:414] "containerMap: removing container" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.322953 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.322973 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.322980 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323026 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323034 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323054 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323061 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323271 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="801546db-f697-46a3-a7df-cc99ae52131b" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323284 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4dca4a6-8732-4d46-970c-f5db40b95df9" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323302 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c800c0aa-716d-40aa-8709-70274d48d9d8" containerName="istio-proxy" Nov 02 08:35:39.325163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.323315 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3674ca-ed1d-4b6a-aba7-8d5202a4d2dd" containerName="istio-proxy" Nov 02 08:35:39.395868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.395836 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc"] Nov 02 08:35:39.396228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.396206 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.443877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.443824 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b"] Nov 02 08:35:39.511427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.511385 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.596216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.583767 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cba6f9af-dockercfg-4jvw5\"" Nov 02 08:35:39.596216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.583939 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cba6f9af\"" Nov 02 08:35:39.596216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.591254 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95prf\" (UniqueName: \"kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.596216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.591569 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frptd\" (UniqueName: \"kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd\") pod \"payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.596216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.592207 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.596216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.592278 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.702451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.702401 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95prf\" (UniqueName: \"kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.702664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.702511 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frptd\" (UniqueName: \"kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd\") pod \"payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.702759 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.702733 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.702829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.702778 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.703858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.703832 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.720472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.720444 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.720752 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.720567 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b"] Nov 02 08:35:39.757874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.757838 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95prf\" (UniqueName: \"kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:39.758163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.757883 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frptd\" (UniqueName: \"kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd\") pod \"payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.805407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.805373 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v"] Nov 02 08:35:39.808665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.808647 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:39.858269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.858133 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:35:39.858441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.858278 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:35:39.858441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.858306 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:35:39.858895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.858871 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:35:39.862918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.862890 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v"] Nov 02 08:35:39.885458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.885418 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:39.906630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.906588 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:39.906754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.906687 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:39.906829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:39.906817 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9r7x\" (UniqueName: \"kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.007967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.007925 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.008117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.007999 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.008117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.008090 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9r7x\" (UniqueName: \"kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.009744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.009717 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.011561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.011531 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.048883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.048833 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:40.061790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.061755 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9r7x\" (UniqueName: \"kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x\") pod \"maas-api-revoke-keys-e2e-api-b-a25202-rsr6v\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.128020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.127750 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:40.297125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.297092 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc"] Nov 02 08:35:40.302720 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:35:40.302688 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8649dacf_8185_4422_9869_872b3b545dc8.slice/crio-cface5bda18ede6cb5e66db1feeb3e4ba6e0d31fcf247a27f4a38382d9a770c6 WatchSource:0}: Error finding container cface5bda18ede6cb5e66db1feeb3e4ba6e0d31fcf247a27f4a38382d9a770c6: Status 404 returned error can't find the container with id cface5bda18ede6cb5e66db1feeb3e4ba6e0d31fcf247a27f4a38382d9a770c6 Nov 02 08:35:40.424932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.424900 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b"] Nov 02 08:35:40.426497 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:35:40.426464 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b386f8_6a6e_40e9_909b_4fb8b7da6ad0.slice/crio-a4739e66d4b52c249be4421fa431b38653988ee363b8eb081fb3db06b0825b06 WatchSource:0}: Error finding container a4739e66d4b52c249be4421fa431b38653988ee363b8eb081fb3db06b0825b06: Status 404 returned error can't find the container with id a4739e66d4b52c249be4421fa431b38653988ee363b8eb081fb3db06b0825b06 Nov 02 08:35:40.523903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.523872 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v"] Nov 02 08:35:40.525241 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:35:40.525207 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca66c261_4985_4737_a1c8_5e29d5f89a81.slice/crio-20b4f0aafd63da35f062126800037dfa3357f9d3dd957538b3c38d94b8bd3185 WatchSource:0}: Error finding container 20b4f0aafd63da35f062126800037dfa3357f9d3dd957538b3c38d94b8bd3185: Status 404 returned error can't find the container with id 20b4f0aafd63da35f062126800037dfa3357f9d3dd957538b3c38d94b8bd3185 Nov 02 08:35:40.989642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.989594 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" event={"ID":"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0","Type":"ContainerStarted","Data":"393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890"} Nov 02 08:35:40.990184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.989695 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" event={"ID":"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0","Type":"ContainerStarted","Data":"a4739e66d4b52c249be4421fa431b38653988ee363b8eb081fb3db06b0825b06"} Nov 02 08:35:40.990184 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.989851 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:40.991805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.991776 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" event={"ID":"8649dacf-8185-4422-9869-872b3b545dc8","Type":"ContainerStarted","Data":"55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7"} Nov 02 08:35:40.991956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.991817 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" event={"ID":"8649dacf-8185-4422-9869-872b3b545dc8","Type":"ContainerStarted","Data":"cface5bda18ede6cb5e66db1feeb3e4ba6e0d31fcf247a27f4a38382d9a770c6"} Nov 02 08:35:40.991956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.991865 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:40.993557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.993524 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" event={"ID":"ca66c261-4985-4737-a1c8-5e29d5f89a81","Type":"ContainerStarted","Data":"0dad46638448ca57ff745b17f8f5baca96c7bf25f9bed0144faee99e4249a3cb"} Nov 02 08:35:40.993720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:40.993565 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" event={"ID":"ca66c261-4985-4737-a1c8-5e29d5f89a81","Type":"ContainerStarted","Data":"20b4f0aafd63da35f062126800037dfa3357f9d3dd957538b3c38d94b8bd3185"} Nov 02 08:35:41.098425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:41.098320 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" podStartSLOduration=2.098302854 podStartE2EDuration="2.098302854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:35:41.095482582 +0000 UTC m=+1788.186103002" watchObservedRunningTime="2026-11-02 08:35:41.098302854 +0000 UTC m=+1788.188923278" Nov 02 08:35:41.202620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:41.202566 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" podStartSLOduration=2.202552149 podStartE2EDuration="2.202552149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:35:41.201377109 +0000 UTC m=+1788.291997529" watchObservedRunningTime="2026-11-02 08:35:41.202552149 +0000 UTC m=+1788.293172567" Nov 02 08:35:41.309242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:41.309188 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" podStartSLOduration=2.309171369 podStartE2EDuration="2.309171369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:35:41.307318317 +0000 UTC m=+1788.397938735" watchObservedRunningTime="2026-11-02 08:35:41.309171369 +0000 UTC m=+1788.399791843" Nov 02 08:35:42.002449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:42.002410 5055 generic.go:356] "Generic (PLEG): container finished" podID="ca66c261-4985-4737-a1c8-5e29d5f89a81" containerID="0dad46638448ca57ff745b17f8f5baca96c7bf25f9bed0144faee99e4249a3cb" exitCode=0 Nov 02 08:35:42.002972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:42.002495 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" event={"ID":"ca66c261-4985-4737-a1c8-5e29d5f89a81","Type":"ContainerDied","Data":"0dad46638448ca57ff745b17f8f5baca96c7bf25f9bed0144faee99e4249a3cb"} Nov 02 08:35:43.203345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.203318 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:43.374947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.374860 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca\") pod \"ca66c261-4985-4737-a1c8-5e29d5f89a81\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " Nov 02 08:35:43.374947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.374903 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca\") pod \"ca66c261-4985-4737-a1c8-5e29d5f89a81\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " Nov 02 08:35:43.375239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.375070 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x\" (UniqueName: \"kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x\") pod \"ca66c261-4985-4737-a1c8-5e29d5f89a81\" (UID: \"ca66c261-4985-4737-a1c8-5e29d5f89a81\") " Nov 02 08:35:43.375426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.375396 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca" pod "ca66c261-4985-4737-a1c8-5e29d5f89a81" (UID: "ca66c261-4985-4737-a1c8-5e29d5f89a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:43.375960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.375935 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:43.377845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.377817 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x" pod "ca66c261-4985-4737-a1c8-5e29d5f89a81" (UID: "ca66c261-4985-4737-a1c8-5e29d5f89a81"). InnerVolumeSpecName "kube-api-access-x9r7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:43.378042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.378019 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca" pod "ca66c261-4985-4737-a1c8-5e29d5f89a81" (UID: "ca66c261-4985-4737-a1c8-5e29d5f89a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:43.477124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.477082 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9r7x\" (UniqueName: \"kubernetes.io/projected/ca66c261-4985-4737-a1c8-5e29d5f89a81-kube-api-access-x9r7x\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:43.477124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.477119 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca66c261-4985-4737-a1c8-5e29d5f89a81-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:43.657277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.657190 5055 patch_prober.go:29] interesting pod/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:35:43.657277 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:43.657255 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:35:44.023078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.023041 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" Nov 02 08:35:44.023078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.023060 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v" event={"ID":"ca66c261-4985-4737-a1c8-5e29d5f89a81","Type":"ContainerDied","Data":"20b4f0aafd63da35f062126800037dfa3357f9d3dd957538b3c38d94b8bd3185"} Nov 02 08:35:44.023428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.023122 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b4f0aafd63da35f062126800037dfa3357f9d3dd957538b3c38d94b8bd3185" Nov 02 08:35:44.701830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.701795 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:35:44.802955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.802856 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.802955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.802894 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.802955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.802941 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.803278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803014 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.803278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:44.803278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803243 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.803431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803331 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.803431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803385 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.803431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803398 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:44.803593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803564 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:44.803879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803855 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.803992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.803898 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data\") pod \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\" (UID: \"765bdaab-c78e-4491-83f1-2c0bc4f4d510\") " Nov 02 08:35:44.804269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804240 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:44.804269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804257 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:44.804688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804662 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.804768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804694 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.804768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804709 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.804768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804723 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.804768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.804737 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.805688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.805659 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:35:44.805909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.805886 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:44.805990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.805886 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:44.806516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.806494 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2" pod "765bdaab-c78e-4491-83f1-2c0bc4f4d510" (UID: "765bdaab-c78e-4491-83f1-2c0bc4f4d510"). InnerVolumeSpecName "kube-api-access-xhtx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:44.906027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.905981 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.906027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.906023 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtx2\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-kube-api-access-xhtx2\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.906315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.906039 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:44.906315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:44.906058 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765bdaab-c78e-4491-83f1-2c0bc4f4d510-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:45.032873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.032830 5055 generic.go:356] "Generic (PLEG): container finished" podID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerID="838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7" exitCode=0 Nov 02 08:35:45.033028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.032876 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" Nov 02 08:35:45.033028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.032904 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" event={"ID":"765bdaab-c78e-4491-83f1-2c0bc4f4d510","Type":"ContainerDied","Data":"838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7"} Nov 02 08:35:45.033028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.032980 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s" event={"ID":"765bdaab-c78e-4491-83f1-2c0bc4f4d510","Type":"ContainerDied","Data":"3f060d55671c4d3a59191bcabe1fb0c1b704f6b31782acbcb7ded11d00645365"} Nov 02 08:35:45.033028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.033003 5055 scope.go:122] "RemoveContainer" containerID="838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7" Nov 02 08:35:45.048536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.048515 5055 scope.go:122] "RemoveContainer" containerID="24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59" Nov 02 08:35:45.059403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.059382 5055 scope.go:122] "RemoveContainer" containerID="838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7" Nov 02 08:35:45.059729 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:35:45.059708 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7\": container with ID starting with 838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7 not found: ID does not exist" containerID="838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7" Nov 02 08:35:45.059780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.059740 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7"} err="failed to get container status \"838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7\": rpc error: code = NotFound desc = could not find container \"838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7\": container with ID starting with 838ce582f70ddbb4fd373086ffc596f6c00bbc23783e229f63f68bbc3aed13a7 not found: ID does not exist" Nov 02 08:35:45.059780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.059770 5055 scope.go:122] "RemoveContainer" containerID="24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59" Nov 02 08:35:45.060062 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:35:45.060038 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59\": container with ID starting with 24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59 not found: ID does not exist" containerID="24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59" Nov 02 08:35:45.060105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.060076 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59"} err="failed to get container status \"24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59\": rpc error: code = NotFound desc = could not find container \"24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59\": container with ID starting with 24672e19bb6e4fafc5275912043521b12e46a27415afddbc05cd3020e16aee59 not found: ID does not exist" Nov 02 08:35:45.187898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.187856 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s"] Nov 02 08:35:45.233749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:45.233714 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f5bb4f-openshift-default-5f5c77b87-h4b9s"] Nov 02 08:35:47.095601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:47.095561 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" path="/var/lib/kubelet/pods/765bdaab-c78e-4491-83f1-2c0bc4f4d510/volumes" Nov 02 08:35:52.018506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:52.018476 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:35:52.019284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:52.019262 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:35:53.678643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:53.678593 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:35:53.690265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:53.690232 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:35:54.512169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:54.499714 5055 scope.go:122] "RemoveContainer" containerID="b904997135230f5454ab8b762e2ba6df011497f9101ff1168c7be48998f74b52" Nov 02 08:35:54.540166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:54.535490 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d"] Nov 02 08:35:54.540166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:54.535750 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" containerID="cri-o://399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a" gracePeriod=30 Nov 02 08:35:54.709459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:54.709430 5055 scope.go:122] "RemoveContainer" containerID="c426cc900aab91fa99982a2d5dd6168faa189dc012dafd92a59ccada6ace090a" Nov 02 08:35:54.764224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:54.764199 5055 scope.go:122] "RemoveContainer" containerID="29d2d9abdab8b69e4b83f8b445c4f2ac7c45ba2dcd7fb9ed51f788f59558cc89" Nov 02 08:35:54.776661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:54.776639 5055 scope.go:122] "RemoveContainer" containerID="8495f278397afd3b5da554a3e8ddac6b0a8616b5b7c093d4e2be7a68229a3ae7" Nov 02 08:35:54.980272 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:35:54.980199 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:35:57.855984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.855938 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6"] Nov 02 08:35:57.856514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.856213 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" containerID="cri-o://5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719" gracePeriod=30 Nov 02 08:35:57.859901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.859872 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw"] Nov 02 08:35:57.860532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.860441 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" containerID="cri-o://d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09" gracePeriod=30 Nov 02 08:35:57.950559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.950523 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl"] Nov 02 08:35:57.951631 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.951607 5055 cpu_manager.go:414] "containerMap: removing container" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" Nov 02 08:35:57.951778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.951768 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" Nov 02 08:35:57.951844 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.951837 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ca66c261-4985-4737-a1c8-5e29d5f89a81" containerName="revoke-keys" Nov 02 08:35:57.951889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.951883 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca66c261-4985-4737-a1c8-5e29d5f89a81" containerName="revoke-keys" Nov 02 08:35:57.952191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.952173 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" Nov 02 08:35:57.952325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.952314 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca66c261-4985-4737-a1c8-5e29d5f89a81" containerName="revoke-keys" Nov 02 08:35:57.952428 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.952418 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" Nov 02 08:35:57.959287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.959258 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:57.970268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.970242 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl"] Nov 02 08:35:57.972245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.972215 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:57.972347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.972293 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:57.972390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:57.972373 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7cg\" (UniqueName: \"kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.013385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.012032 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:35:58.023360 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.023320 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:35:58.023638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.023491 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:35:58.073477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.073441 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.073676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.073531 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7cg\" (UniqueName: \"kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.073824 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.073801 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.074569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.074542 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.077883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.077852 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.113683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.113514 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:35:58.126828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.126793 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7cg\" (UniqueName: \"kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg\") pod \"maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.314100 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.314046 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:35:58.756743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:58.756704 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl"] Nov 02 08:35:58.756994 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:35:58.756970 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f801fc3_bad7_44eb_b816_aa912fb745c1.slice/crio-b2c60efa88cb91dc0679fc51457cfaf306cf12b8902465517b4cfe1718c9222b WatchSource:0}: Error finding container b2c60efa88cb91dc0679fc51457cfaf306cf12b8902465517b4cfe1718c9222b: Status 404 returned error can't find the container with id b2c60efa88cb91dc0679fc51457cfaf306cf12b8902465517b4cfe1718c9222b Nov 02 08:35:59.555097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.553878 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" event={"ID":"3f801fc3-bad7-44eb-b816-aa912fb745c1","Type":"ContainerStarted","Data":"21c2dc59393c69d8f0d69fb49da6c42336bbad2184c340128ea60c76ea1b1822"} Nov 02 08:35:59.555097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.553956 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" event={"ID":"3f801fc3-bad7-44eb-b816-aa912fb745c1","Type":"ContainerStarted","Data":"b2c60efa88cb91dc0679fc51457cfaf306cf12b8902465517b4cfe1718c9222b"} Nov 02 08:35:59.666222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.665967 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" podStartSLOduration=2.665943511 podStartE2EDuration="2.665943511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:35:59.66171275 +0000 UTC m=+1806.752333168" watchObservedRunningTime="2026-11-02 08:35:59.665943511 +0000 UTC m=+1806.756563930" Nov 02 08:35:59.869088 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.868722 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:35:59.945997 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.945959 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946019 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946074 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946111 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946152 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946189 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946393 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946431 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.946538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.946465 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs\") pod \"ff52f76f-736a-4fb7-9500-186ffbaa1063\" (UID: \"ff52f76f-736a-4fb7-9500-186ffbaa1063\") " Nov 02 08:35:59.948189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.948040 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.948468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.948355 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.948899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.948610 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.948899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.948861 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:59.949128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.949094 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.949781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.949754 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "kube-api-access-nhszs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:59.950579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.950551 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:35:59.952541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.952506 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.952849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:35:59.952821 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token" pod "ff52f76f-736a-4fb7-9500-186ffbaa1063" (UID: "ff52f76f-736a-4fb7-9500-186ffbaa1063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048858 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048890 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048901 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff52f76f-736a-4fb7-9500-186ffbaa1063-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048910 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhszs\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-kube-api-access-nhszs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048922 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048932 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048942 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff52f76f-736a-4fb7-9500-186ffbaa1063-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048950 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.049327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.048958 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff52f76f-736a-4fb7-9500-186ffbaa1063-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.564642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.564589 5055 generic.go:356] "Generic (PLEG): container finished" podID="3f801fc3-bad7-44eb-b816-aa912fb745c1" containerID="21c2dc59393c69d8f0d69fb49da6c42336bbad2184c340128ea60c76ea1b1822" exitCode=0 Nov 02 08:36:00.565113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.564665 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" event={"ID":"3f801fc3-bad7-44eb-b816-aa912fb745c1","Type":"ContainerDied","Data":"21c2dc59393c69d8f0d69fb49da6c42336bbad2184c340128ea60c76ea1b1822"} Nov 02 08:36:00.566453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.566422 5055 generic.go:356] "Generic (PLEG): container finished" podID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerID="399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a" exitCode=0 Nov 02 08:36:00.566600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.566474 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" Nov 02 08:36:00.566600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.566507 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" event={"ID":"ff52f76f-736a-4fb7-9500-186ffbaa1063","Type":"ContainerDied","Data":"399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a"} Nov 02 08:36:00.566600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.566564 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d" event={"ID":"ff52f76f-736a-4fb7-9500-186ffbaa1063","Type":"ContainerDied","Data":"625b48f2dbbbae212c8fe13cc7f4f1affb5f630db875ad17c5e1c7bdecbc5153"} Nov 02 08:36:00.566600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.566593 5055 scope.go:122] "RemoveContainer" containerID="399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a" Nov 02 08:36:00.581117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.580997 5055 scope.go:122] "RemoveContainer" containerID="399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a" Nov 02 08:36:00.581462 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:00.581436 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a\": container with ID starting with 399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a not found: ID does not exist" containerID="399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a" Nov 02 08:36:00.581553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.581475 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a"} err="failed to get container status \"399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a\": rpc error: code = NotFound desc = could not find container \"399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a\": container with ID starting with 399a8bc4e430d75901aae97b01495e657a853855a609a7532a263a1d24de797a not found: ID does not exist" Nov 02 08:36:00.794700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.794665 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d"] Nov 02 08:36:00.801215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:00.801175 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-bb97d"] Nov 02 08:36:01.096483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.096445 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" path="/var/lib/kubelet/pods/ff52f76f-736a-4fb7-9500-186ffbaa1063/volumes" Nov 02 08:36:01.825188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.825122 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:36:01.880033 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.879955 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca\") pod \"3f801fc3-bad7-44eb-b816-aa912fb745c1\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " Nov 02 08:36:01.880241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.880112 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca\") pod \"3f801fc3-bad7-44eb-b816-aa912fb745c1\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " Nov 02 08:36:01.880241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.880181 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg\" (UniqueName: \"kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg\") pod \"3f801fc3-bad7-44eb-b816-aa912fb745c1\" (UID: \"3f801fc3-bad7-44eb-b816-aa912fb745c1\") " Nov 02 08:36:01.880613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.880578 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca" pod "3f801fc3-bad7-44eb-b816-aa912fb745c1" (UID: "3f801fc3-bad7-44eb-b816-aa912fb745c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:01.881063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.881037 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:01.883817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.883775 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg" pod "3f801fc3-bad7-44eb-b816-aa912fb745c1" (UID: "3f801fc3-bad7-44eb-b816-aa912fb745c1"). InnerVolumeSpecName "kube-api-access-fz7cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:01.884532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.884497 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca" pod "3f801fc3-bad7-44eb-b816-aa912fb745c1" (UID: "3f801fc3-bad7-44eb-b816-aa912fb745c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:01.983006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.982964 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f801fc3-bad7-44eb-b816-aa912fb745c1-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:01.983006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:01.983007 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7cg\" (UniqueName: \"kubernetes.io/projected/3f801fc3-bad7-44eb-b816-aa912fb745c1-kube-api-access-fz7cg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:02.591172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:02.590695 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" event={"ID":"3f801fc3-bad7-44eb-b816-aa912fb745c1","Type":"ContainerDied","Data":"b2c60efa88cb91dc0679fc51457cfaf306cf12b8902465517b4cfe1718c9222b"} Nov 02 08:36:02.591172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:02.590759 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2c60efa88cb91dc0679fc51457cfaf306cf12b8902465517b4cfe1718c9222b" Nov 02 08:36:02.591172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:02.590822 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl" Nov 02 08:36:02.911979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:02.911382 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:36:02.911979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:02.911482 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:03.228103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.228068 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.318825 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.318946 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319020 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319071 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319181 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319204 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319262 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319300 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.319332 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data\") pod \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\" (UID: \"7f9120df-a37d-4d38-b323-4ed820bbd0a0\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.320063 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.320401 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.320450 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.320574 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.320613 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.322476 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.322706 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.322768 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "kube-api-access-wsx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.323053 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo" pod "7f9120df-a37d-4d38-b323-4ed820bbd0a0" (UID: "7f9120df-a37d-4d38-b323-4ed820bbd0a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.331824 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.420969 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421015 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421049 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421092 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421160 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421187 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421228 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421388 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.421650 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token\") pod \"882efdab-68b3-4b74-8a06-4760a14a669d\" (UID: \"882efdab-68b3-4b74-8a06-4760a14a669d\") " Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422563 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422595 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/7f9120df-a37d-4d38-b323-4ed820bbd0a0-kube-api-access-wsx7d\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422612 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422626 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422643 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422658 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422673 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422690 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9120df-a37d-4d38-b323-4ed820bbd0a0-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.422705 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9120df-a37d-4d38-b323-4ed820bbd0a0-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.423583 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.423813 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.424286 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.424584 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.428320 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.429760 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.429911 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.430576 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.431117 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p" pod "882efdab-68b3-4b74-8a06-4760a14a669d" (UID: "882efdab-68b3-4b74-8a06-4760a14a669d"). InnerVolumeSpecName "kube-api-access-h878p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529318 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529344 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529354 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529365 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529373 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882efdab-68b3-4b74-8a06-4760a14a669d-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529381 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882efdab-68b3-4b74-8a06-4760a14a669d-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529391 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h878p\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-kube-api-access-h878p\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529400 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882efdab-68b3-4b74-8a06-4760a14a669d-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.558572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.529408 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882efdab-68b3-4b74-8a06-4760a14a669d-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:03.604389 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.604346 5055 generic.go:356] "Generic (PLEG): container finished" podID="882efdab-68b3-4b74-8a06-4760a14a669d" containerID="d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09" exitCode=0 Nov 02 08:36:03.604389 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.604395 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" Nov 02 08:36:03.604735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.604428 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" event={"ID":"882efdab-68b3-4b74-8a06-4760a14a669d","Type":"ContainerDied","Data":"d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09"} Nov 02 08:36:03.604735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.604513 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw" event={"ID":"882efdab-68b3-4b74-8a06-4760a14a669d","Type":"ContainerDied","Data":"998bf55e7db7bd5389ee35c99071de65b31575355cb56491c90b7dc57293191d"} Nov 02 08:36:03.604735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.604547 5055 scope.go:122] "RemoveContainer" containerID="d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09" Nov 02 08:36:03.607072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.607040 5055 generic.go:356] "Generic (PLEG): container finished" podID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerID="5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719" exitCode=0 Nov 02 08:36:03.607281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.607127 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" event={"ID":"7f9120df-a37d-4d38-b323-4ed820bbd0a0","Type":"ContainerDied","Data":"5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719"} Nov 02 08:36:03.607364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.607325 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" event={"ID":"7f9120df-a37d-4d38-b323-4ed820bbd0a0","Type":"ContainerDied","Data":"3300d13e0fa7d8d95078969f73d122dee0dccbae74527e9aed49385c861fe5d3"} Nov 02 08:36:03.607435 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.607420 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6" Nov 02 08:36:03.639664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.639617 5055 scope.go:122] "RemoveContainer" containerID="d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09" Nov 02 08:36:03.640525 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:03.640484 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09\": container with ID starting with d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09 not found: ID does not exist" containerID="d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09" Nov 02 08:36:03.640655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.640537 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09"} err="failed to get container status \"d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09\": rpc error: code = NotFound desc = could not find container \"d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09\": container with ID starting with d7b63eeed299d588acb4e6bd28e3c6142d304f6a3234f95f381bb9f6b903ab09 not found: ID does not exist" Nov 02 08:36:03.640655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.640563 5055 scope.go:122] "RemoveContainer" containerID="5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719" Nov 02 08:36:03.658397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.658365 5055 scope.go:122] "RemoveContainer" containerID="178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf" Nov 02 08:36:03.669891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.669864 5055 scope.go:122] "RemoveContainer" containerID="5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719" Nov 02 08:36:03.670285 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:03.670262 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719\": container with ID starting with 5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719 not found: ID does not exist" containerID="5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719" Nov 02 08:36:03.670346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.670297 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719"} err="failed to get container status \"5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719\": rpc error: code = NotFound desc = could not find container \"5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719\": container with ID starting with 5ab1c3a6d5c4416d2cc4bc6774e188699bb5804105e9053ccc608bc2d0d77719 not found: ID does not exist" Nov 02 08:36:03.670346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.670317 5055 scope.go:122] "RemoveContainer" containerID="178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf" Nov 02 08:36:03.670564 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:03.670548 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf\": container with ID starting with 178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf not found: ID does not exist" containerID="178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf" Nov 02 08:36:03.670608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.670567 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf"} err="failed to get container status \"178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf\": rpc error: code = NotFound desc = could not find container \"178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf\": container with ID starting with 178acb2d0ec09141145965d81d6957a3c14a4e4d6691fa9edeb01cfaddb3b4bf not found: ID does not exist" Nov 02 08:36:03.762595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.762537 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6"] Nov 02 08:36:03.767464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.767422 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-kbmp6"] Nov 02 08:36:03.868753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.868656 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw"] Nov 02 08:36:03.872355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:03.872325 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-phwrw"] Nov 02 08:36:04.661112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:04.661079 5055 generic.go:356] "Generic (PLEG): container finished" podID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerID="5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b" exitCode=0 Nov 02 08:36:04.662819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:04.661165 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" event={"ID":"048aa870-d17c-47ee-9ed0-583b3324f9d8","Type":"ContainerDied","Data":"5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b"} Nov 02 08:36:04.662819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:04.661593 5055 scope.go:122] "RemoveContainer" containerID="5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b" Nov 02 08:36:04.663394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:04.663354 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:04.663452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:04.663437 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:04.663495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:04.663482 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:05.095746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.095704 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" path="/var/lib/kubelet/pods/7f9120df-a37d-4d38-b323-4ed820bbd0a0/volumes" Nov 02 08:36:05.096542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.096518 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" path="/var/lib/kubelet/pods/882efdab-68b3-4b74-8a06-4760a14a669d/volumes" Nov 02 08:36:05.374468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374125 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb"] Nov 02 08:36:05.374948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374923 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" Nov 02 08:36:05.374948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374947 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374968 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3f801fc3-bad7-44eb-b816-aa912fb745c1" containerName="revoke-keys" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374976 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f801fc3-bad7-44eb-b816-aa912fb745c1" containerName="revoke-keys" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374990 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.374997 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.375037 5055 cpu_manager.go:414] "containerMap: removing container" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.375045 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.375060 5055 cpu_manager.go:414] "containerMap: removing container" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" Nov 02 08:36:05.375229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.375067 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765bdaab-c78e-4491-83f1-2c0bc4f4d510" containerName="istio-proxy" Nov 02 08:36:05.380492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.380462 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="882efdab-68b3-4b74-8a06-4760a14a669d" containerName="istio-proxy" Nov 02 08:36:05.380733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.380718 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" Nov 02 08:36:05.380857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.380844 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff52f76f-736a-4fb7-9500-186ffbaa1063" containerName="istio-proxy" Nov 02 08:36:05.380959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.380949 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f801fc3-bad7-44eb-b816-aa912fb745c1" containerName="revoke-keys" Nov 02 08:36:05.381056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.381045 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" Nov 02 08:36:05.399463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.399429 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.458556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.458520 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb"] Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582351 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582410 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582520 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582556 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582600 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582623 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582725 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582789 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jr8f\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.584166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.582834 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687236 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687300 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687420 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687454 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687499 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687524 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687641 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687709 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jr8f\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.688321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.687758 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.689259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.688355 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.690220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.689636 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.690220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.689901 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.702647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.695866 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.702647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.696167 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.736157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.730870 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.758667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.758626 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.760061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.759543 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:05.766849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.766821 5055 generic.go:356] "Generic (PLEG): container finished" podID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerID="b19ce90bee1516120ebeb15cffcc6a2dd9a2b340ed901c8996490bac28a574b2" exitCode=0 Nov 02 08:36:05.767011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.766916 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" event={"ID":"f374ffb5-7bb8-4663-9afe-f1a528ea2069","Type":"ContainerDied","Data":"b19ce90bee1516120ebeb15cffcc6a2dd9a2b340ed901c8996490bac28a574b2"} Nov 02 08:36:05.770780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.769603 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" event={"ID":"048aa870-d17c-47ee-9ed0-583b3324f9d8","Type":"ContainerStarted","Data":"3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656"} Nov 02 08:36:05.770780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.769655 5055 scope.go:122] "RemoveContainer" containerID="b19ce90bee1516120ebeb15cffcc6a2dd9a2b340ed901c8996490bac28a574b2" Nov 02 08:36:05.771948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.771908 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:05.772066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.772010 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:05.772206 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.772063 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:05.775689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:05.775606 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jr8f\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:06.044109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.043816 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:06.515229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.515184 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb"] Nov 02 08:36:06.520711 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:36:06.520675 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83556cc2_9236_4f14_b20c_81a9ed57bdc2.slice/crio-65a2bbe2ce61b03d4203974726893e05192530120318f01cdf9d7dba08b27405 WatchSource:0}: Error finding container 65a2bbe2ce61b03d4203974726893e05192530120318f01cdf9d7dba08b27405: Status 404 returned error can't find the container with id 65a2bbe2ce61b03d4203974726893e05192530120318f01cdf9d7dba08b27405 Nov 02 08:36:06.524285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.524238 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:06.524423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.524342 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:06.524423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.524388 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:06.610580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.610546 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:36:06.610918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.610893 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:36:06.611068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.611041 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 08:36:06.779511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.779409 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" event={"ID":"f374ffb5-7bb8-4663-9afe-f1a528ea2069","Type":"ContainerStarted","Data":"847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5"} Nov 02 08:36:06.793164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.785350 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" event={"ID":"83556cc2-9236-4f14-b20c-81a9ed57bdc2","Type":"ContainerStarted","Data":"80bb848f054b317ec4a4e0f3107f61c0588b69b096c057a58ef89850a364f67f"} Nov 02 08:36:06.793164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.785421 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" event={"ID":"83556cc2-9236-4f14-b20c-81a9ed57bdc2","Type":"ContainerStarted","Data":"65a2bbe2ce61b03d4203974726893e05192530120318f01cdf9d7dba08b27405"} Nov 02 08:36:06.963163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:06.963057 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:36:07.048315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.048210 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:07.048500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.048351 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:07.048500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.048409 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:07.049953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.049897 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podStartSLOduration=2.049883579 podStartE2EDuration="2.049883579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:07.036611761 +0000 UTC m=+1814.127232193" watchObservedRunningTime="2026-11-02 08:36:07.049883579 +0000 UTC m=+1814.140503998" Nov 02 08:36:07.516167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.515763 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:36:07.530161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.527324 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:07.530161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.527442 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:07.612182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.611599 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:36:07.612182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:07.611663 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 08:36:08.137372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:08.137322 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:08.178362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:08.141193 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:08.530165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:08.529284 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:08.530165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:08.529373 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:08.652176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:08.637640 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:36:08.652176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:08.637751 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 08:36:09.078166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:09.077396 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:09.078166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:09.077472 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:09.521169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:09.520168 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:09.521169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:09.520253 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:09.658845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:09.658791 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:36:09.659160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:09.659110 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 08:36:10.078166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.051573 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:10.078498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.078454 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:10.245165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.228117 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.245165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.228470 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.264315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.251793 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.264315 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.253911 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.302165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.280217 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.378469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.378363 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" podUID="5d13bf2a-d174-4b83-b363-ed42f963cff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.527864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.527822 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:10.528477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.528442 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:10.602177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.599419 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:10.608369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.608332 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:36:10.608632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.608605 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 08:36:10.608770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.608754 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 02 08:36:10.957166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.953347 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 08:36:10.965775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:10.962268 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.170160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.142272 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:11.170160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.142360 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:11.244254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.236536 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.299166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.270938 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.306770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.274182 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.307372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.285366 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.308876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.285417 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.310618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.310589 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:36:11.310901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.310886 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:36:11.311186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.311173 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:36:11.311562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.311549 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:36:11.355162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.335856 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.537235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.515801 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.537235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.515948 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 08:36:11.577176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.575128 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:11.577176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.575211 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:11.775397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.775340 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.842438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.842325 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:11.869563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.866644 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:36:11.895166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.888480 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:36:11.895166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.888554 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 08:36:11.962617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.961898 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:36:11.964275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.964244 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 08:36:11.969708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:11.964938 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:36:12.126256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.094129 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 08:36:12.152493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.142388 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:12.152493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.142490 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:12.152493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.142650 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:36:12.271871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.271815 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 08:36:12.376866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.360263 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:12.376866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.363288 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:12.376866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.368096 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 08:36:12.377333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.377301 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:36:12.377433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.377417 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:36:12.507162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.461969 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:12.507162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.466544 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k" podUID="5d13bf2a-d174-4b83-b363-ed42f963cff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:12.555164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.521536 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:12.555164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.521600 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:12.608162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.604280 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:36:12.765573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.758484 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:36:12.859975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.859933 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:36:12.860402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.860276 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:36:12.860635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.860612 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:36:12.860784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.860771 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:36:12.860914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.860901 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:36:12.861554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.861527 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:36:12.862591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.862556 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:36:12.878682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.878653 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:36:12.879357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.879328 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:36:12.920166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:12.919513 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:36:13.070400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:13.070306 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:13.072021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:13.071980 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:13.532234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:13.520511 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 08:36:13.532234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:13.520577 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 08:36:14.044518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:14.044482 5055 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:36:14.045019 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:14.044546 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 08:36:14.525609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:14.525568 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:36:14.538486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:14.538446 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:36:15.053213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:15.053107 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:15.054916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:15.054584 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:15.056280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:15.056241 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:36:16.093940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:16.093895 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c"] Nov 02 08:36:16.138927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:16.138891 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26f03-blp7c"] Nov 02 08:36:17.093913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:17.093876 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb781d3-f800-4d77-b187-4dd63e51f331" path="/var/lib/kubelet/pods/5eb781d3-f800-4d77-b187-4dd63e51f331/volumes" Nov 02 08:36:22.696164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.692965 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2"] Nov 02 08:36:22.696164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.693963 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" Nov 02 08:36:22.696164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.693986 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9120df-a37d-4d38-b323-4ed820bbd0a0" containerName="istio-proxy" Nov 02 08:36:22.726157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.726107 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.788223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.787399 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45fc31-openshift-default-dockercfg-l4pq9\"" Nov 02 08:36:22.836555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.836515 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2"] Nov 02 08:36:22.889120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.889080 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.889511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.889481 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbx9\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.889791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.889752 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.890042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.890026 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.890238 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.890222 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.890422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.890406 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.890547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.890533 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.890762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.890741 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.890906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.890889 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.992979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.992945 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993015 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993083 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993111 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993201 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993229 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993342 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993387 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbx9\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993417 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.993686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.993524 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.994118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.994067 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.994118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.994095 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.994335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.994195 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.994618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.994596 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.996194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.996162 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:22.996194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:22.996179 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:23.049643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.049595 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:23.049801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.049661 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbx9\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9\") pod \"e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:23.085674 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.085632 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:23.519963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.519933 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2"] Nov 02 08:36:23.520128 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:36:23.520093 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108ddb47_d760_4bc5_97cc_25716febd86a.slice/crio-d0bc79f1b7d0b3592ccd26856ba4139d5f1a6094467bd5f72a6770665bc5520a WatchSource:0}: Error finding container d0bc79f1b7d0b3592ccd26856ba4139d5f1a6094467bd5f72a6770665bc5520a: Status 404 returned error can't find the container with id d0bc79f1b7d0b3592ccd26856ba4139d5f1a6094467bd5f72a6770665bc5520a Nov 02 08:36:23.523094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.523046 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:23.523242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.523193 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:23.523299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:23.523247 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:36:24.284094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.284052 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" event={"ID":"108ddb47-d760-4bc5-97cc-25716febd86a","Type":"ContainerStarted","Data":"ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36"} Nov 02 08:36:24.284569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.284169 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" event={"ID":"108ddb47-d760-4bc5-97cc-25716febd86a","Type":"ContainerStarted","Data":"d0bc79f1b7d0b3592ccd26856ba4139d5f1a6094467bd5f72a6770665bc5520a"} Nov 02 08:36:24.394720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.394661 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podStartSLOduration=2.394646093 podStartE2EDuration="2.394646093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:24.391717733 +0000 UTC m=+1831.482338151" watchObservedRunningTime="2026-11-02 08:36:24.394646093 +0000 UTC m=+1831.485266511" Nov 02 08:36:24.488313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.488261 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx"] Nov 02 08:36:24.492715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.492686 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.518921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.518875 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.519232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.519002 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.519232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.519041 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdfnt\" (UniqueName: \"kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.541960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.541850 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:36:24.541960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.541855 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:36:24.541960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.541872 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:36:24.541960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.541902 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:36:24.548909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.548867 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx"] Nov 02 08:36:24.623655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.623620 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.623862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.623723 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.623862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.623755 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdfnt\" (UniqueName: \"kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.625242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.625130 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.627013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.626986 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.677423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.677381 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdfnt\" (UniqueName: \"kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt\") pod \"maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.808566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:24.808474 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:24.934095 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:24.934045 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:36:25.094412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:25.094326 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:25.094622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:25.094449 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:25.183776 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:25.183732 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx"] Nov 02 08:36:25.184448 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:36:25.184415 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f306175_0983_47cb_81f7_4d7acace5fc7.slice/crio-1064911296eb84718a5e3ac91ef170111f3c25fcd721fc40336fc87560a5d13a WatchSource:0}: Error finding container 1064911296eb84718a5e3ac91ef170111f3c25fcd721fc40336fc87560a5d13a: Status 404 returned error can't find the container with id 1064911296eb84718a5e3ac91ef170111f3c25fcd721fc40336fc87560a5d13a Nov 02 08:36:25.295124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:25.295087 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" event={"ID":"9f306175-0983-47cb-81f7-4d7acace5fc7","Type":"ContainerStarted","Data":"1064911296eb84718a5e3ac91ef170111f3c25fcd721fc40336fc87560a5d13a"} Nov 02 08:36:25.295633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:25.295543 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:25.296515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:25.296492 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:36:26.306280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:26.306236 5055 generic.go:356] "Generic (PLEG): container finished" podID="9f306175-0983-47cb-81f7-4d7acace5fc7" containerID="86aed61b14f1a250385ab5540c7b04a184d8f3fd8bba2ed2ed1e23aaa6564499" exitCode=0 Nov 02 08:36:26.306753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:26.306323 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" event={"ID":"9f306175-0983-47cb-81f7-4d7acace5fc7","Type":"ContainerDied","Data":"86aed61b14f1a250385ab5540c7b04a184d8f3fd8bba2ed2ed1e23aaa6564499"} Nov 02 08:36:27.502223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.502190 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:27.573175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.573058 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt\" (UniqueName: \"kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt\") pod \"9f306175-0983-47cb-81f7-4d7acace5fc7\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " Nov 02 08:36:27.573175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.573166 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca\") pod \"9f306175-0983-47cb-81f7-4d7acace5fc7\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " Nov 02 08:36:27.573394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.573231 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca\") pod \"9f306175-0983-47cb-81f7-4d7acace5fc7\" (UID: \"9f306175-0983-47cb-81f7-4d7acace5fc7\") " Nov 02 08:36:27.573684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.573657 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca" pod "9f306175-0983-47cb-81f7-4d7acace5fc7" (UID: "9f306175-0983-47cb-81f7-4d7acace5fc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:27.573860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.573843 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:27.575449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.575422 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca" pod "9f306175-0983-47cb-81f7-4d7acace5fc7" (UID: "9f306175-0983-47cb-81f7-4d7acace5fc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:27.575449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.575436 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt" pod "9f306175-0983-47cb-81f7-4d7acace5fc7" (UID: "9f306175-0983-47cb-81f7-4d7acace5fc7"). InnerVolumeSpecName "kube-api-access-rdfnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:27.675555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.675513 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdfnt\" (UniqueName: \"kubernetes.io/projected/9f306175-0983-47cb-81f7-4d7acace5fc7-kube-api-access-rdfnt\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:27.675555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:27.675553 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f306175-0983-47cb-81f7-4d7acace5fc7-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:28.323274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:28.323235 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" Nov 02 08:36:28.323445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:28.323275 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx" event={"ID":"9f306175-0983-47cb-81f7-4d7acace5fc7","Type":"ContainerDied","Data":"1064911296eb84718a5e3ac91ef170111f3c25fcd721fc40336fc87560a5d13a"} Nov 02 08:36:28.323445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:28.323332 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1064911296eb84718a5e3ac91ef170111f3c25fcd721fc40336fc87560a5d13a" Nov 02 08:36:30.690164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.680200 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn"] Nov 02 08:36:30.690164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.688594 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9f306175-0983-47cb-81f7-4d7acace5fc7" containerName="revoke-keys" Nov 02 08:36:30.690164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.688619 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f306175-0983-47cb-81f7-4d7acace5fc7" containerName="revoke-keys" Nov 02 08:36:30.690164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.688724 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f306175-0983-47cb-81f7-4d7acace5fc7" containerName="revoke-keys" Nov 02 08:36:30.692712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.692684 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.739326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.739284 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn"] Nov 02 08:36:30.741797 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.741763 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqx9\" (UniqueName: \"kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9\") pod \"payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.741960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.741941 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.789213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.788419 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45fc31\"" Nov 02 08:36:30.789213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.788573 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45fc31-dockercfg-r78h9\"" Nov 02 08:36:30.858166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.843120 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqx9\" (UniqueName: \"kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9\") pod \"payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.982027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.920290 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l"] Nov 02 08:36:30.985879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.985845 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.987552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.987522 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.989125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.989103 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:30.997376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.997322 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqx9\" (UniqueName: \"kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9\") pod \"payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:30.999285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:30.999258 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l"] Nov 02 08:36:31.009746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.009703 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:31.088544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.088222 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrwf4\" (UniqueName: \"kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4\") pod \"payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.088544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.088404 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.189940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.189893 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrwf4\" (UniqueName: \"kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4\") pod \"payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.190130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.190094 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.191096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.191062 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.244135 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.244062 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrwf4\" (UniqueName: \"kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4\") pod \"payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.343134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.343098 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:31.417907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.417442 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn"] Nov 02 08:36:31.433608 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:36:31.433576 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22056cb3_e9d3_4c56_9a3f_b08ab1196ab1.slice/crio-65902549d1f46159fdef70ab20ad22470697985b5ca6d84c155469304932098a WatchSource:0}: Error finding container 65902549d1f46159fdef70ab20ad22470697985b5ca6d84c155469304932098a: Status 404 returned error can't find the container with id 65902549d1f46159fdef70ab20ad22470697985b5ca6d84c155469304932098a Nov 02 08:36:31.718076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:31.718022 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l"] Nov 02 08:36:31.720863 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:36:31.720828 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c62f0b5_26f5_4f5d_ba28_1e4966374af6.slice/crio-b7f0ee019f201162b61efcc2ef1cf9db73f44c4f7f216037db05756d6f6d8e6a WatchSource:0}: Error finding container b7f0ee019f201162b61efcc2ef1cf9db73f44c4f7f216037db05756d6f6d8e6a: Status 404 returned error can't find the container with id b7f0ee019f201162b61efcc2ef1cf9db73f44c4f7f216037db05756d6f6d8e6a Nov 02 08:36:32.363672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.363634 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" event={"ID":"9c62f0b5-26f5-4f5d-ba28-1e4966374af6","Type":"ContainerStarted","Data":"18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572"} Nov 02 08:36:32.363845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.363695 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" event={"ID":"9c62f0b5-26f5-4f5d-ba28-1e4966374af6","Type":"ContainerStarted","Data":"b7f0ee019f201162b61efcc2ef1cf9db73f44c4f7f216037db05756d6f6d8e6a"} Nov 02 08:36:32.363927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.363847 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:32.365510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.365482 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" event={"ID":"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1","Type":"ContainerStarted","Data":"26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650"} Nov 02 08:36:32.365654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.365529 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" event={"ID":"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1","Type":"ContainerStarted","Data":"65902549d1f46159fdef70ab20ad22470697985b5ca6d84c155469304932098a"} Nov 02 08:36:32.365654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.365577 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:32.481037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.480960 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" podStartSLOduration=2.4809394129999998 podStartE2EDuration="2.480939413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:32.471275862 +0000 UTC m=+1839.561896282" watchObservedRunningTime="2026-11-02 08:36:32.480939413 +0000 UTC m=+1839.571559835" Nov 02 08:36:32.580936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:32.580851 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" podStartSLOduration=2.580831072 podStartE2EDuration="2.580831072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:32.579311703 +0000 UTC m=+1839.669932125" watchObservedRunningTime="2026-11-02 08:36:32.580831072 +0000 UTC m=+1839.671451499" Nov 02 08:36:33.062789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:33.062746 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874"] Nov 02 08:36:33.063338 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:33.063054 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" containerID="cri-o://289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f" gracePeriod=30 Nov 02 08:36:37.589959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.589863 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r"] Nov 02 08:36:37.595801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.595775 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.649265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.649231 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r"] Nov 02 08:36:37.690546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.689987 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:36:37.690546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.690010 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:36:37.690546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.690460 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:36:37.691195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.691176 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:36:37.746321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.746277 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvhg\" (UniqueName: \"kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.746552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.746345 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.746552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.746409 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.848627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.848537 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvhg\" (UniqueName: \"kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.848627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.848588 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.848627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.848623 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.849404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.849379 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.851053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.851033 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.902084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.902033 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvhg\" (UniqueName: \"kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:37.911759 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:37.911727 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:38.293895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.293665 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r"] Nov 02 08:36:38.349048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.349021 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:36:38.358336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358269 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358345 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358437 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358461 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358479 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358505 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358529 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:38.358772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358584 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358620 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:38.358772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358713 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:38.358772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358718 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358872 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data\") pod \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\" (UID: \"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a\") " Nov 02 08:36:38.358990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.358907 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:38.359183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.359154 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:38.359643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.359618 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.359743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.359648 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.359743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.359665 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.359743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.359679 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.359743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.359693 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.360952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.360930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:36:38.361031 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.360979 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:38.361279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.361256 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:36:38.361371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.361285 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz" pod "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" (UID: "aafdd2c8-572f-4ff1-ade7-431fb9d39a9a"). InnerVolumeSpecName "kube-api-access-ch7dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:38.439349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.439315 5055 generic.go:356] "Generic (PLEG): container finished" podID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerID="289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f" exitCode=0 Nov 02 08:36:38.439526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.439370 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" Nov 02 08:36:38.439526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.439410 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" event={"ID":"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a","Type":"ContainerDied","Data":"289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f"} Nov 02 08:36:38.439526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.439465 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874" event={"ID":"aafdd2c8-572f-4ff1-ade7-431fb9d39a9a","Type":"ContainerDied","Data":"18bf9da9d016c5ccb45a32c7aa770fce1903474980ab7c3df9fe29d47e9aa409"} Nov 02 08:36:38.439526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.439489 5055 scope.go:122] "RemoveContainer" containerID="289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f" Nov 02 08:36:38.441049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.441028 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" event={"ID":"87cca28c-6e1c-4b75-996c-8c6734a92dad","Type":"ContainerStarted","Data":"e19faec4dea4179e98ed6840fde709aff4ac5186d81e4b824ca2d06a121a890f"} Nov 02 08:36:38.453550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.452254 5055 scope.go:122] "RemoveContainer" containerID="717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c" Nov 02 08:36:38.461668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.461639 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.461834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.461672 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7dz\" (UniqueName: \"kubernetes.io/projected/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-kube-api-access-ch7dz\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.461834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.461687 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.461834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.461701 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:38.466476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.466448 5055 scope.go:122] "RemoveContainer" containerID="289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f" Nov 02 08:36:38.466870 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:38.466836 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f\": container with ID starting with 289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f not found: ID does not exist" containerID="289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f" Nov 02 08:36:38.466968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.466874 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f"} err="failed to get container status \"289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f\": rpc error: code = NotFound desc = could not find container \"289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f\": container with ID starting with 289647e113c41260f58e39efe3931b1278acf101b0fc2d51731ff90d965cd93f not found: ID does not exist" Nov 02 08:36:38.466968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.466902 5055 scope.go:122] "RemoveContainer" containerID="717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c" Nov 02 08:36:38.467266 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:38.467224 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c\": container with ID starting with 717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c not found: ID does not exist" containerID="717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c" Nov 02 08:36:38.467376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.467274 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c"} err="failed to get container status \"717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c\": rpc error: code = NotFound desc = could not find container \"717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c\": container with ID starting with 717ebccd722c7140579f0ced3a86fdc516c25ca44b7560dcfda3b26ebaa7076c not found: ID does not exist" Nov 02 08:36:38.567127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.567088 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874"] Nov 02 08:36:38.570953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:38.570924 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b26f03-openshift-default-7d5d7795c5-2f874"] Nov 02 08:36:39.099062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:39.099024 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" path="/var/lib/kubelet/pods/aafdd2c8-572f-4ff1-ade7-431fb9d39a9a/volumes" Nov 02 08:36:39.453507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:39.453470 5055 generic.go:356] "Generic (PLEG): container finished" podID="87cca28c-6e1c-4b75-996c-8c6734a92dad" containerID="a919a391ca022b682286eab734e12fa2f14f11d73f4a285c4fec86128029ca9a" exitCode=0 Nov 02 08:36:39.453701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:39.453551 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" event={"ID":"87cca28c-6e1c-4b75-996c-8c6734a92dad","Type":"ContainerDied","Data":"a919a391ca022b682286eab734e12fa2f14f11d73f4a285c4fec86128029ca9a"} Nov 02 08:36:40.639149 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.639120 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:40.696784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.696711 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca\") pod \"87cca28c-6e1c-4b75-996c-8c6734a92dad\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " Nov 02 08:36:40.696994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.696799 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca\") pod \"87cca28c-6e1c-4b75-996c-8c6734a92dad\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " Nov 02 08:36:40.696994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.696846 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg\" (UniqueName: \"kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg\") pod \"87cca28c-6e1c-4b75-996c-8c6734a92dad\" (UID: \"87cca28c-6e1c-4b75-996c-8c6734a92dad\") " Nov 02 08:36:40.697252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.697220 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca" pod "87cca28c-6e1c-4b75-996c-8c6734a92dad" (UID: "87cca28c-6e1c-4b75-996c-8c6734a92dad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:40.697619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.697595 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:40.699223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.699195 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg" pod "87cca28c-6e1c-4b75-996c-8c6734a92dad" (UID: "87cca28c-6e1c-4b75-996c-8c6734a92dad"). InnerVolumeSpecName "kube-api-access-xnvhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:40.699405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.699386 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca" pod "87cca28c-6e1c-4b75-996c-8c6734a92dad" (UID: "87cca28c-6e1c-4b75-996c-8c6734a92dad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:40.798893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.798858 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cca28c-6e1c-4b75-996c-8c6734a92dad-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:40.798893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:40.798890 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnvhg\" (UniqueName: \"kubernetes.io/projected/87cca28c-6e1c-4b75-996c-8c6734a92dad-kube-api-access-xnvhg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:41.359394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:41.359298 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:36:41.528576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:41.526835 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" event={"ID":"87cca28c-6e1c-4b75-996c-8c6734a92dad","Type":"ContainerDied","Data":"e19faec4dea4179e98ed6840fde709aff4ac5186d81e4b824ca2d06a121a890f"} Nov 02 08:36:41.528576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:41.526902 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19faec4dea4179e98ed6840fde709aff4ac5186d81e4b824ca2d06a121a890f" Nov 02 08:36:41.528576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:41.526967 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r" Nov 02 08:36:43.397837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.397806 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:36:43.398328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.397881 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:36:43.895926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.895877 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s"] Nov 02 08:36:43.896685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896666 5055 cpu_manager.go:414] "containerMap: removing container" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" Nov 02 08:36:43.896742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896687 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" Nov 02 08:36:43.896742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896719 5055 cpu_manager.go:414] "containerMap: removing container" podUID="87cca28c-6e1c-4b75-996c-8c6734a92dad" containerName="revoke-keys" Nov 02 08:36:43.896742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896724 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cca28c-6e1c-4b75-996c-8c6734a92dad" containerName="revoke-keys" Nov 02 08:36:43.896845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896829 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" Nov 02 08:36:43.896845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896844 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cca28c-6e1c-4b75-996c-8c6734a92dad" containerName="revoke-keys" Nov 02 08:36:43.896910 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.896851 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" Nov 02 08:36:43.901269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.901232 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:43.949765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.949722 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:36:43.950000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.949841 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:36:43.950000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.949881 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:36:43.950000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.949978 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:36:43.955045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.955003 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s"] Nov 02 08:36:43.957210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.957174 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:43.957550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.957519 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrs5\" (UniqueName: \"kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:43.957694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:43.957604 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.059180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.059098 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.059382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.059256 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrs5\" (UniqueName: \"kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.059382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.059313 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.060025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.059997 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.061620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.061596 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.113309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.113274 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrs5\" (UniqueName: \"kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5\") pod \"maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.218217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.218122 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:44.608207 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:36:44.608175 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b6e323_079f_4bde_801a_1470f6b74f13.slice/crio-c294e8a5e78b6750cb75872a560ab3098e2d8f2607e87845502787a9156fd4a4 WatchSource:0}: Error finding container c294e8a5e78b6750cb75872a560ab3098e2d8f2607e87845502787a9156fd4a4: Status 404 returned error can't find the container with id c294e8a5e78b6750cb75872a560ab3098e2d8f2607e87845502787a9156fd4a4 Nov 02 08:36:44.608553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:44.608214 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s"] Nov 02 08:36:45.560734 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:45.560698 5055 generic.go:356] "Generic (PLEG): container finished" podID="b6b6e323-079f-4bde-801a-1470f6b74f13" containerID="530ccce5d61ef1d9372df18ebc8f67ed3237af00f12464b3188b10b3dbe7b919" exitCode=0 Nov 02 08:36:45.560851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:45.560781 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" event={"ID":"b6b6e323-079f-4bde-801a-1470f6b74f13","Type":"ContainerDied","Data":"530ccce5d61ef1d9372df18ebc8f67ed3237af00f12464b3188b10b3dbe7b919"} Nov 02 08:36:45.560916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:45.560858 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" event={"ID":"b6b6e323-079f-4bde-801a-1470f6b74f13","Type":"ContainerStarted","Data":"c294e8a5e78b6750cb75872a560ab3098e2d8f2607e87845502787a9156fd4a4"} Nov 02 08:36:46.749457 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.749433 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:46.900553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.900434 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca\") pod \"b6b6e323-079f-4bde-801a-1470f6b74f13\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " Nov 02 08:36:46.900553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.900504 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5\" (UniqueName: \"kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5\") pod \"b6b6e323-079f-4bde-801a-1470f6b74f13\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " Nov 02 08:36:46.900777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.900618 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca\") pod \"b6b6e323-079f-4bde-801a-1470f6b74f13\" (UID: \"b6b6e323-079f-4bde-801a-1470f6b74f13\") " Nov 02 08:36:46.900892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.900868 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca" pod "b6b6e323-079f-4bde-801a-1470f6b74f13" (UID: "b6b6e323-079f-4bde-801a-1470f6b74f13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:46.901461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.901432 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:46.902926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.902896 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5" pod "b6b6e323-079f-4bde-801a-1470f6b74f13" (UID: "b6b6e323-079f-4bde-801a-1470f6b74f13"). InnerVolumeSpecName "kube-api-access-kfrs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:46.903278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:46.903252 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca" pod "b6b6e323-079f-4bde-801a-1470f6b74f13" (UID: "b6b6e323-079f-4bde-801a-1470f6b74f13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:47.003274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:47.003238 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrs5\" (UniqueName: \"kubernetes.io/projected/b6b6e323-079f-4bde-801a-1470f6b74f13-kube-api-access-kfrs5\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:47.003274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:47.003271 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6e323-079f-4bde-801a-1470f6b74f13-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:47.577627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:47.577597 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" Nov 02 08:36:47.577627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:47.577602 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s" event={"ID":"b6b6e323-079f-4bde-801a-1470f6b74f13","Type":"ContainerDied","Data":"c294e8a5e78b6750cb75872a560ab3098e2d8f2607e87845502787a9156fd4a4"} Nov 02 08:36:47.577862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:36:47.577657 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c294e8a5e78b6750cb75872a560ab3098e2d8f2607e87845502787a9156fd4a4" Nov 02 08:36:54.920888 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:36:54.920848 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:37:14.261442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:14.261395 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx"] Nov 02 08:37:14.307384 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:14.307338 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bca68b-b28lx"] Nov 02 08:37:15.096004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:15.095964 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f306175-0983-47cb-81f7-4d7acace5fc7" path="/var/lib/kubelet/pods/9f306175-0983-47cb-81f7-4d7acace5fc7/volumes" Nov 02 08:37:21.594308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.594270 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4"] Nov 02 08:37:21.595354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.595330 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b6b6e323-079f-4bde-801a-1470f6b74f13" containerName="revoke-keys" Nov 02 08:37:21.595417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.595358 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b6e323-079f-4bde-801a-1470f6b74f13" containerName="revoke-keys" Nov 02 08:37:21.595417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.595376 5055 cpu_manager.go:414] "containerMap: removing container" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" Nov 02 08:37:21.595417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.595384 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafdd2c8-572f-4ff1-ade7-431fb9d39a9a" containerName="istio-proxy" Nov 02 08:37:21.595572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.595559 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b6e323-079f-4bde-801a-1470f6b74f13" containerName="revoke-keys" Nov 02 08:37:21.600080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.600056 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.651158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.651111 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4"] Nov 02 08:37:21.693217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.692227 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:37:21.693217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.692376 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:37:21.694443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.694416 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:37:21.694620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.694583 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:37:21.760076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.760037 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lmr\" (UniqueName: \"kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.760076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.760072 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.760295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.760274 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.861395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.861304 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.861554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.861486 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lmr\" (UniqueName: \"kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.861554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.861514 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.862258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.862230 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.863893 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.863857 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.914868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.914829 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lmr\" (UniqueName: \"kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr\") pod \"maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:21.915115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:21.915099 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:22.296708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:22.296613 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4"] Nov 02 08:37:22.296875 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:37:22.296770 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954ba9ae_6da4_4b3c_a84c_216539f0207a.slice/crio-e19ad6041b476b04d83c5489a52659e5088ad4b60e737817f51608a78dc01529 WatchSource:0}: Error finding container e19ad6041b476b04d83c5489a52659e5088ad4b60e737817f51608a78dc01529: Status 404 returned error can't find the container with id e19ad6041b476b04d83c5489a52659e5088ad4b60e737817f51608a78dc01529 Nov 02 08:37:22.985232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:22.985190 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" event={"ID":"954ba9ae-6da4-4b3c-a84c-216539f0207a","Type":"ContainerStarted","Data":"5033f4c6b928793cd4a397d8f19432b1d4d7788cfc0763bee2c7ca2252c0031d"} Nov 02 08:37:22.985713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:22.985262 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" event={"ID":"954ba9ae-6da4-4b3c-a84c-216539f0207a","Type":"ContainerStarted","Data":"e19ad6041b476b04d83c5489a52659e5088ad4b60e737817f51608a78dc01529"} Nov 02 08:37:23.093808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:23.093751 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" podStartSLOduration=2.093735976 podStartE2EDuration="2.093735976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:23.091126841 +0000 UTC m=+1890.181747261" watchObservedRunningTime="2026-11-02 08:37:23.093735976 +0000 UTC m=+1890.184356395" Nov 02 08:37:23.999089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:23.999047 5055 generic.go:356] "Generic (PLEG): container finished" podID="954ba9ae-6da4-4b3c-a84c-216539f0207a" containerID="5033f4c6b928793cd4a397d8f19432b1d4d7788cfc0763bee2c7ca2252c0031d" exitCode=0 Nov 02 08:37:23.999582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:23.999197 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" event={"ID":"954ba9ae-6da4-4b3c-a84c-216539f0207a","Type":"ContainerDied","Data":"5033f4c6b928793cd4a397d8f19432b1d4d7788cfc0763bee2c7ca2252c0031d"} Nov 02 08:37:25.007273 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:37:25.007225 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:37:25.181926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.181896 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:25.220254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.220221 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca\") pod \"954ba9ae-6da4-4b3c-a84c-216539f0207a\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " Nov 02 08:37:25.220414 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.220288 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca\") pod \"954ba9ae-6da4-4b3c-a84c-216539f0207a\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " Nov 02 08:37:25.220461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.220435 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr\" (UniqueName: \"kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr\") pod \"954ba9ae-6da4-4b3c-a84c-216539f0207a\" (UID: \"954ba9ae-6da4-4b3c-a84c-216539f0207a\") " Nov 02 08:37:25.220649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.220623 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca" pod "954ba9ae-6da4-4b3c-a84c-216539f0207a" (UID: "954ba9ae-6da4-4b3c-a84c-216539f0207a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:25.221006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.220983 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:25.222544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.222512 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr" pod "954ba9ae-6da4-4b3c-a84c-216539f0207a" (UID: "954ba9ae-6da4-4b3c-a84c-216539f0207a"). InnerVolumeSpecName "kube-api-access-47lmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:25.222655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.222548 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca" pod "954ba9ae-6da4-4b3c-a84c-216539f0207a" (UID: "954ba9ae-6da4-4b3c-a84c-216539f0207a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:25.322920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.322816 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ba9ae-6da4-4b3c-a84c-216539f0207a-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:25.322920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:25.322863 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47lmr\" (UniqueName: \"kubernetes.io/projected/954ba9ae-6da4-4b3c-a84c-216539f0207a-kube-api-access-47lmr\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:26.023632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:26.023599 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" Nov 02 08:37:26.024087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:26.023604 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4" event={"ID":"954ba9ae-6da4-4b3c-a84c-216539f0207a","Type":"ContainerDied","Data":"e19ad6041b476b04d83c5489a52659e5088ad4b60e737817f51608a78dc01529"} Nov 02 08:37:26.024087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:26.023730 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19ad6041b476b04d83c5489a52659e5088ad4b60e737817f51608a78dc01529" Nov 02 08:37:26.051917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:26.051880 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v"] Nov 02 08:37:26.103286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:26.103243 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a25202-rsr6v"] Nov 02 08:37:27.140723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:27.140681 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca66c261-4985-4737-a1c8-5e29d5f89a81" path="/var/lib/kubelet/pods/ca66c261-4985-4737-a1c8-5e29d5f89a81/volumes" Nov 02 08:37:34.840983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:34.840942 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r"] Nov 02 08:37:34.914257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:34.910223 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ee94-brw2r"] Nov 02 08:37:35.093563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:35.093477 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cca28c-6e1c-4b75-996c-8c6734a92dad" path="/var/lib/kubelet/pods/87cca28c-6e1c-4b75-996c-8c6734a92dad/volumes" Nov 02 08:37:39.005382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:39.005333 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4"] Nov 02 08:37:39.048990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:39.048943 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c17125-j54g4"] Nov 02 08:37:39.094546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:39.094493 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954ba9ae-6da4-4b3c-a84c-216539f0207a" path="/var/lib/kubelet/pods/954ba9ae-6da4-4b3c-a84c-216539f0207a/volumes" Nov 02 08:37:39.115687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:39.115644 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal" podUID="34d28ca7a91586006c95d24bde05028b" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:37:41.478440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:41.478373 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr"] Nov 02 08:37:41.478981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:41.478658 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" containerID="cri-o://e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452" gracePeriod=30 Nov 02 08:37:41.842916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:41.842806 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:37:41.842916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:41.842876 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:37:43.239279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.239242 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78"] Nov 02 08:37:43.240081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.240061 5055 cpu_manager.go:414] "containerMap: removing container" podUID="954ba9ae-6da4-4b3c-a84c-216539f0207a" containerName="revoke-keys" Nov 02 08:37:43.240182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.240082 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954ba9ae-6da4-4b3c-a84c-216539f0207a" containerName="revoke-keys" Nov 02 08:37:43.240261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.240249 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="954ba9ae-6da4-4b3c-a84c-216539f0207a" containerName="revoke-keys" Nov 02 08:37:43.245633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.245614 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.300929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.300887 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78"] Nov 02 08:37:43.314752 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.314710 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.315001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.314967 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwp8v\" (UniqueName: \"kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.315275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.315252 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.338880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.338849 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:37:43.339046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.338947 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:37:43.339046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.339013 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:37:43.342404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.342383 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:37:43.417190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.417155 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.417403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.417258 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.417478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.417418 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwp8v\" (UniqueName: \"kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.417988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.417963 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.419731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.419711 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.470501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.470463 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwp8v\" (UniqueName: \"kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.560117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.560029 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:43.944634 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:37:43.944605 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bef17e4_39d0_4528_99e8_64eff2f5d710.slice/crio-8e36cd94005df0be034942946ab2697746eae3b309df5b813d4fa1eb5ebeb025 WatchSource:0}: Error finding container 8e36cd94005df0be034942946ab2697746eae3b309df5b813d4fa1eb5ebeb025: Status 404 returned error can't find the container with id 8e36cd94005df0be034942946ab2697746eae3b309df5b813d4fa1eb5ebeb025 Nov 02 08:37:43.945760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:43.945733 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78"] Nov 02 08:37:44.184179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.184078 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" event={"ID":"3bef17e4-39d0-4528-99e8-64eff2f5d710","Type":"ContainerStarted","Data":"8e36cd94005df0be034942946ab2697746eae3b309df5b813d4fa1eb5ebeb025"} Nov 02 08:37:44.647294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.647196 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m"] Nov 02 08:37:44.651843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.651813 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:44.706023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.705988 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m"] Nov 02 08:37:44.757862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.750401 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d51175-dockercfg-5kwtj\"" Nov 02 08:37:44.767229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.765339 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d51175\"" Nov 02 08:37:44.774176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.767843 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdcgp\" (UniqueName: \"kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp\") pod \"payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:44.779796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.774457 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:44.905492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.905407 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99"] Nov 02 08:37:44.918937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.918034 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:44.939183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.936839 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdcgp\" (UniqueName: \"kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp\") pod \"payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:44.939183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.936910 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:44.939183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.938018 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:44.939183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.917927 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99"] Nov 02 08:37:44.991836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:44.991799 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdcgp\" (UniqueName: \"kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp\") pod \"payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:45.038571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.038533 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc2l\" (UniqueName: \"kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l\") pod \"payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.038777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.038663 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.140686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.140651 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc2l\" (UniqueName: \"kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l\") pod \"payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.140840 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.140759 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.141468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.141448 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.194200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.194110 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc2l\" (UniqueName: \"kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l\") pod \"payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.194528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.194495 5055 generic.go:356] "Generic (PLEG): container finished" podID="3bef17e4-39d0-4528-99e8-64eff2f5d710" containerID="4fc4c904ec4bb50c7b79e54a6048ad742c9f1b0386d618dfdb8683ec651fc758" exitCode=0 Nov 02 08:37:45.194659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.194598 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" event={"ID":"3bef17e4-39d0-4528-99e8-64eff2f5d710","Type":"ContainerDied","Data":"4fc4c904ec4bb50c7b79e54a6048ad742c9f1b0386d618dfdb8683ec651fc758"} Nov 02 08:37:45.256449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.256409 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:45.267051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.267020 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:45.637378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.637342 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99"] Nov 02 08:37:45.638591 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:37:45.638559 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be91159_3cb5_4572_a325_33eba9d83460.slice/crio-1fa3ec8db9536a1bb6d41701a6956c8788358825e76b10264fc53acc125b75eb WatchSource:0}: Error finding container 1fa3ec8db9536a1bb6d41701a6956c8788358825e76b10264fc53acc125b75eb: Status 404 returned error can't find the container with id 1fa3ec8db9536a1bb6d41701a6956c8788358825e76b10264fc53acc125b75eb Nov 02 08:37:45.686023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:45.685992 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m"] Nov 02 08:37:46.204439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.204399 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" event={"ID":"4be91159-3cb5-4572-a325-33eba9d83460","Type":"ContainerStarted","Data":"0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6"} Nov 02 08:37:46.204651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.204478 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:46.204651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.204499 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" event={"ID":"4be91159-3cb5-4572-a325-33eba9d83460","Type":"ContainerStarted","Data":"1fa3ec8db9536a1bb6d41701a6956c8788358825e76b10264fc53acc125b75eb"} Nov 02 08:37:46.206258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.206215 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" event={"ID":"c04e4ecf-187b-4f5e-a92c-589ba7f17888","Type":"ContainerStarted","Data":"547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072"} Nov 02 08:37:46.206393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.206270 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" event={"ID":"c04e4ecf-187b-4f5e-a92c-589ba7f17888","Type":"ContainerStarted","Data":"f00c53ee0fcd9fa9500fb01b4e9c77b510286e60c6b64355b17270b97a7b71c6"} Nov 02 08:37:46.206393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.206335 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:46.315477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.315406 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" podStartSLOduration=2.315386171 podStartE2EDuration="2.315386171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:46.310587024 +0000 UTC m=+1913.401207444" watchObservedRunningTime="2026-11-02 08:37:46.315386171 +0000 UTC m=+1913.406006591" Nov 02 08:37:46.362276 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.362250 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:46.420273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.420218 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" podStartSLOduration=2.420202149 podStartE2EDuration="2.420202149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:46.419063743 +0000 UTC m=+1913.509684162" watchObservedRunningTime="2026-11-02 08:37:46.420202149 +0000 UTC m=+1913.510822567" Nov 02 08:37:46.464215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.464107 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca\") pod \"3bef17e4-39d0-4528-99e8-64eff2f5d710\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " Nov 02 08:37:46.464396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.464378 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca\") pod \"3bef17e4-39d0-4528-99e8-64eff2f5d710\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " Nov 02 08:37:46.464550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.464531 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v\" (UniqueName: \"kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v\") pod \"3bef17e4-39d0-4528-99e8-64eff2f5d710\" (UID: \"3bef17e4-39d0-4528-99e8-64eff2f5d710\") " Nov 02 08:37:46.465188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.465126 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca" pod "3bef17e4-39d0-4528-99e8-64eff2f5d710" (UID: "3bef17e4-39d0-4528-99e8-64eff2f5d710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:46.471531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.471488 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca" pod "3bef17e4-39d0-4528-99e8-64eff2f5d710" (UID: "3bef17e4-39d0-4528-99e8-64eff2f5d710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:46.472984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.472951 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v" pod "3bef17e4-39d0-4528-99e8-64eff2f5d710" (UID: "3bef17e4-39d0-4528-99e8-64eff2f5d710"). InnerVolumeSpecName "kube-api-access-cwp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:46.570862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.570803 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwp8v\" (UniqueName: \"kubernetes.io/projected/3bef17e4-39d0-4528-99e8-64eff2f5d710-kube-api-access-cwp8v\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.571119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.571095 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.571268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.571252 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef17e4-39d0-4528-99e8-64eff2f5d710-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.805892 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.805861 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:37:46.878158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878116 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878186 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878325 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878358 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878394 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878437 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:46.878568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878460 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878489 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:46.878568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878517 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878650 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878750 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx\") pod \"0e503d1b-7474-4901-94ac-978681e017cc\" (UID: \"0e503d1b-7474-4901-94ac-978681e017cc\") " Nov 02 08:37:46.878877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878785 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:46.878932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878905 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:46.878991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.878957 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:46.879419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.879390 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.879419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.879419 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.879640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.879436 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e503d1b-7474-4901-94ac-978681e017cc-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.879640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.879450 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.879640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.879465 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.880650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.880621 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:46.880905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.880884 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:46.880952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.880915 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "kube-api-access-2cxwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:46.881233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.881205 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo" pod "0e503d1b-7474-4901-94ac-978681e017cc" (UID: "0e503d1b-7474-4901-94ac-978681e017cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:37:46.981341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.981305 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.981341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.981337 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e503d1b-7474-4901-94ac-978681e017cc-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.981341 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.981348 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxwx\" (UniqueName: \"kubernetes.io/projected/0e503d1b-7474-4901-94ac-978681e017cc-kube-api-access-2cxwx\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:46.981584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:46.981358 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e503d1b-7474-4901-94ac-978681e017cc-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:47.216562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.216516 5055 generic.go:356] "Generic (PLEG): container finished" podID="0e503d1b-7474-4901-94ac-978681e017cc" containerID="e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452" exitCode=0 Nov 02 08:37:47.216754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.216556 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" event={"ID":"0e503d1b-7474-4901-94ac-978681e017cc","Type":"ContainerDied","Data":"e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452"} Nov 02 08:37:47.216754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.216622 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" event={"ID":"0e503d1b-7474-4901-94ac-978681e017cc","Type":"ContainerDied","Data":"f9a934af8fbdcc865e1e48b10eb4f5d7f8f51437a944b723c2d3c4e470894373"} Nov 02 08:37:47.216754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.216566 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr" Nov 02 08:37:47.216754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.216646 5055 scope.go:122] "RemoveContainer" containerID="e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452" Nov 02 08:37:47.219190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.218724 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" event={"ID":"3bef17e4-39d0-4528-99e8-64eff2f5d710","Type":"ContainerDied","Data":"8e36cd94005df0be034942946ab2697746eae3b309df5b813d4fa1eb5ebeb025"} Nov 02 08:37:47.219190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.218784 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e36cd94005df0be034942946ab2697746eae3b309df5b813d4fa1eb5ebeb025" Nov 02 08:37:47.219190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.218835 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78" Nov 02 08:37:47.229087 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.229067 5055 scope.go:122] "RemoveContainer" containerID="8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3" Nov 02 08:37:47.239052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.239028 5055 scope.go:122] "RemoveContainer" containerID="e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452" Nov 02 08:37:47.239449 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:37:47.239420 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452\": container with ID starting with e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452 not found: ID does not exist" containerID="e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452" Nov 02 08:37:47.239594 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.239458 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452"} err="failed to get container status \"e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452\": rpc error: code = NotFound desc = could not find container \"e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452\": container with ID starting with e78e8906cfeba9e03ecafa8507259015bb9b00004bced8f81f4d7e36e0cc5452 not found: ID does not exist" Nov 02 08:37:47.239594 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.239478 5055 scope.go:122] "RemoveContainer" containerID="8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3" Nov 02 08:37:47.239789 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:37:47.239767 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3\": container with ID starting with 8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3 not found: ID does not exist" containerID="8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3" Nov 02 08:37:47.239853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.239797 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3"} err="failed to get container status \"8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3\": rpc error: code = NotFound desc = could not find container \"8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3\": container with ID starting with 8f7cc2afbc5c5d226ac390cf1020794ec8dff073112c240ad7ece465dde374c3 not found: ID does not exist" Nov 02 08:37:47.373084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.373004 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr"] Nov 02 08:37:47.418692 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:47.418653 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-19ee94-openshift-default-78dc4bb6f7-rsvpr"] Nov 02 08:37:49.095330 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:49.095287 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e503d1b-7474-4901-94ac-978681e017cc" path="/var/lib/kubelet/pods/0e503d1b-7474-4901-94ac-978681e017cc/volumes" Nov 02 08:37:54.925865 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:37:54.925814 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:37:55.397891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.397849 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x"] Nov 02 08:37:55.398680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398656 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" Nov 02 08:37:55.398680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398680 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" Nov 02 08:37:55.398838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398704 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3bef17e4-39d0-4528-99e8-64eff2f5d710" containerName="revoke-keys" Nov 02 08:37:55.398838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398712 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bef17e4-39d0-4528-99e8-64eff2f5d710" containerName="revoke-keys" Nov 02 08:37:55.398838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398735 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" Nov 02 08:37:55.398838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398744 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" Nov 02 08:37:55.398965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398903 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" Nov 02 08:37:55.398965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398912 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bef17e4-39d0-4528-99e8-64eff2f5d710" containerName="revoke-keys" Nov 02 08:37:55.398965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.398922 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e503d1b-7474-4901-94ac-978681e017cc" containerName="istio-proxy" Nov 02 08:37:55.404736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.404713 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534494 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534642 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cp5m\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534714 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534792 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534874 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534951 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.534992 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.535115 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.535560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.535173 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646122 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646238 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646317 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646356 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646465 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646501 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646527 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646638 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cp5m\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.646705 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.647208 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.647704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.647469 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.649515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.648329 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.649515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.648569 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.649515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.649183 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.704163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.704099 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x"] Nov 02 08:37:55.728510 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.727263 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.752206 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.748525 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.780699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.780656 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:55.909256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:55.897457 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cp5m\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:56.020954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:56.020905 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:56.489109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:56.489073 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x"] Nov 02 08:37:56.495758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:56.495184 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:37:56.495758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:56.495277 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:37:56.495758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:56.495321 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:37:57.223119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:57.223086 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:37:57.223695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:57.223192 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:37:57.308891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:57.308792 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" event={"ID":"a11b9c5e-69f8-41a8-b476-142a0945882e","Type":"ContainerStarted","Data":"22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93"} Nov 02 08:37:57.308891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:57.308858 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" event={"ID":"a11b9c5e-69f8-41a8-b476-142a0945882e","Type":"ContainerStarted","Data":"bdf5ca7f352b660680d3e0f6a055a61150fbfceed2fc3dba40e17da7496a1844"} Nov 02 08:37:57.551295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:57.551181 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podStartSLOduration=2.551165052 podStartE2EDuration="2.551165052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:57.548814121 +0000 UTC m=+1924.639434541" watchObservedRunningTime="2026-11-02 08:37:57.551165052 +0000 UTC m=+1924.641785464" Nov 02 08:37:58.022194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:58.022124 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:37:58.023672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:58.023641 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 08:37:58.023888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:58.023865 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 08:37:59.022173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:59.022111 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 08:37:59.022691 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:37:59.022185 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 08:38:00.021732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.021692 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 08:38:00.021964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.021791 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 08:38:00.317843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.317748 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt"] Nov 02 08:38:00.323506 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.323474 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.361377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.361340 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt"] Nov 02 08:38:00.362010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.361982 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.362179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.362019 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.362258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.362234 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzgn\" (UniqueName: \"kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.396164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.372814 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:38:00.396164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.372826 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:38:00.411165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.405330 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:38:00.411165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.405456 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:38:00.463092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.463055 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzgn\" (UniqueName: \"kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.463296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.463244 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.463296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.463270 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.463950 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.463927 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.467525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.467496 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.518042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.518004 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzgn\" (UniqueName: \"kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn\") pod \"maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:00.642679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:00.642591 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:01.021449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:01.021406 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 08:38:01.021664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:01.021471 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 08:38:01.047357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:01.047318 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt"] Nov 02 08:38:01.048212 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:38:01.048183 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da564fc_50a5_46f2_ada7_e33ffe2f8d43.slice/crio-04d59811b0fcc7814ce0666ef6b7efe7106023c4c4994f91c7088f4b7cf905d6 WatchSource:0}: Error finding container 04d59811b0fcc7814ce0666ef6b7efe7106023c4c4994f91c7088f4b7cf905d6: Status 404 returned error can't find the container with id 04d59811b0fcc7814ce0666ef6b7efe7106023c4c4994f91c7088f4b7cf905d6 Nov 02 08:38:01.356174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:01.356105 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" event={"ID":"5da564fc-50a5-46f2-ada7-e33ffe2f8d43","Type":"ContainerStarted","Data":"04d59811b0fcc7814ce0666ef6b7efe7106023c4c4994f91c7088f4b7cf905d6"} Nov 02 08:38:02.021782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:02.021742 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 08:38:02.021977 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:02.021812 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 08:38:02.366512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:02.366465 5055 generic.go:356] "Generic (PLEG): container finished" podID="5da564fc-50a5-46f2-ada7-e33ffe2f8d43" containerID="e7160460d7d9a122c1419c42ca5db20d9710eaae1621fad0ac37f1b0a73c55ef" exitCode=0 Nov 02 08:38:02.366948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:02.366556 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" event={"ID":"5da564fc-50a5-46f2-ada7-e33ffe2f8d43","Type":"ContainerDied","Data":"e7160460d7d9a122c1419c42ca5db20d9710eaae1621fad0ac37f1b0a73c55ef"} Nov 02 08:38:03.022378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.022336 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 08:38:03.022542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.022445 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 08:38:03.578114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.578086 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:03.644856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.644761 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca\") pod \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " Nov 02 08:38:03.645021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.644858 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn\" (UniqueName: \"kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn\") pod \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " Nov 02 08:38:03.645021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.644987 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca\") pod \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\" (UID: \"5da564fc-50a5-46f2-ada7-e33ffe2f8d43\") " Nov 02 08:38:03.645255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.645217 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca" pod "5da564fc-50a5-46f2-ada7-e33ffe2f8d43" (UID: "5da564fc-50a5-46f2-ada7-e33ffe2f8d43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:03.645639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.645618 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:03.647181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.647136 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca" pod "5da564fc-50a5-46f2-ada7-e33ffe2f8d43" (UID: "5da564fc-50a5-46f2-ada7-e33ffe2f8d43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:03.647321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.647297 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn" pod "5da564fc-50a5-46f2-ada7-e33ffe2f8d43" (UID: "5da564fc-50a5-46f2-ada7-e33ffe2f8d43"). InnerVolumeSpecName "kube-api-access-bvzgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:03.747805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.747755 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:03.747805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:03.747800 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvzgn\" (UniqueName: \"kubernetes.io/projected/5da564fc-50a5-46f2-ada7-e33ffe2f8d43-kube-api-access-bvzgn\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:04.027888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:04.027843 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:38:04.028305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:04.028282 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:38:04.029208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:04.029182 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:38:04.389885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:04.389778 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" Nov 02 08:38:04.390094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:04.389778 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt" event={"ID":"5da564fc-50a5-46f2-ada7-e33ffe2f8d43","Type":"ContainerDied","Data":"04d59811b0fcc7814ce0666ef6b7efe7106023c4c4994f91c7088f4b7cf905d6"} Nov 02 08:38:04.390094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:04.389938 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d59811b0fcc7814ce0666ef6b7efe7106023c4c4994f91c7088f4b7cf905d6" Nov 02 08:38:05.760868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:05.760804 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd"] Nov 02 08:38:05.762225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:05.762188 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" containerID="cri-o://dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc" gracePeriod=30 Nov 02 08:38:07.830730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:07.830678 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:38:08.772675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:08.772625 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:38:08.772878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:08.772696 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:38:10.513986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.513936 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5"] Nov 02 08:38:10.514851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.514828 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5da564fc-50a5-46f2-ada7-e33ffe2f8d43" containerName="revoke-keys" Nov 02 08:38:10.514851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.514851 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da564fc-50a5-46f2-ada7-e33ffe2f8d43" containerName="revoke-keys" Nov 02 08:38:10.515006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.514985 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da564fc-50a5-46f2-ada7-e33ffe2f8d43" containerName="revoke-keys" Nov 02 08:38:10.519111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.519085 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633519 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633585 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633715 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633761 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633805 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633895 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k96l\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.633955 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.634011 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.634744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.634121 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.658767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.658729 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5"] Nov 02 08:38:10.735430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735392 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735430 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735439 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735520 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735559 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735603 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735690 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k96l\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735743 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735879 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.735999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.735916 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.736060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.736014 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.736123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.736090 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.736123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.736100 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.736123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.736116 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.736415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.736392 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.738431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.738410 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.738550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.738514 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.807857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.807819 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.809089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.808884 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k96l\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:10.833887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:10.833859 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:11.159529 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.159503 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:38:11.254449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.254415 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.254662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.254482 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.254662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.254517 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.254662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.254563 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.254662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.254611 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.254969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.254940 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:11.255223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255199 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.255368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255338 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:11.255501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255467 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:11.255568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255523 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.255627 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255586 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.255751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255707 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:11.255751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255725 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket\") pod \"be57046b-d6ac-417b-b856-2a940616bd25\" (UID: \"be57046b-d6ac-417b-b856-2a940616bd25\") " Nov 02 08:38:11.256012 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.255988 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:11.256640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.256615 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.256743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.256644 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.256743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.256659 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.256743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.256674 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.256743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.256688 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be57046b-d6ac-417b-b856-2a940616bd25-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.257129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.257109 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:38:11.257129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.257119 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "kube-api-access-xd2pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:11.257250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.257130 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:11.257816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.257799 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token" pod "be57046b-d6ac-417b-b856-2a940616bd25" (UID: "be57046b-d6ac-417b-b856-2a940616bd25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:11.311279 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:38:11.311234 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67882b59_ab1a_4697_890d_616c30dc77df.slice/crio-6d270a3663d69678cad9c8fc4c1b741345d5a1951159d3dc8858250abc62c797 WatchSource:0}: Error finding container 6d270a3663d69678cad9c8fc4c1b741345d5a1951159d3dc8858250abc62c797: Status 404 returned error can't find the container with id 6d270a3663d69678cad9c8fc4c1b741345d5a1951159d3dc8858250abc62c797 Nov 02 08:38:11.315023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.314973 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:11.315285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.315077 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:11.315285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.315126 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:11.316477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.316452 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5"] Nov 02 08:38:11.358540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.358503 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be57046b-d6ac-417b-b856-2a940616bd25-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.358540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.358540 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be57046b-d6ac-417b-b856-2a940616bd25-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.358735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.358553 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2pc\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-kube-api-access-xd2pc\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.358735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.358566 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be57046b-d6ac-417b-b856-2a940616bd25-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:11.469736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.469694 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" event={"ID":"67882b59-ab1a-4697-890d-616c30dc77df","Type":"ContainerStarted","Data":"d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e"} Nov 02 08:38:11.469947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.469769 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" event={"ID":"67882b59-ab1a-4697-890d-616c30dc77df","Type":"ContainerStarted","Data":"6d270a3663d69678cad9c8fc4c1b741345d5a1951159d3dc8858250abc62c797"} Nov 02 08:38:11.472348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.472316 5055 generic.go:356] "Generic (PLEG): container finished" podID="be57046b-d6ac-417b-b856-2a940616bd25" containerID="dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc" exitCode=0 Nov 02 08:38:11.472503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.472361 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" event={"ID":"be57046b-d6ac-417b-b856-2a940616bd25","Type":"ContainerDied","Data":"dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc"} Nov 02 08:38:11.472503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.472366 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" Nov 02 08:38:11.472503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.472459 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd" event={"ID":"be57046b-d6ac-417b-b856-2a940616bd25","Type":"ContainerDied","Data":"377a56ebd577b5b51a3ff8f95300b1b5ea6261a27b0a8ff03b20be4a0a4a2ebb"} Nov 02 08:38:11.472638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.472511 5055 scope.go:122] "RemoveContainer" containerID="dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc" Nov 02 08:38:11.535595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.534525 5055 scope.go:122] "RemoveContainer" containerID="e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1" Nov 02 08:38:11.554733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.554708 5055 scope.go:122] "RemoveContainer" containerID="dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc" Nov 02 08:38:11.555171 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:11.555126 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc\": container with ID starting with dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc not found: ID does not exist" containerID="dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc" Nov 02 08:38:11.555256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.555181 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc"} err="failed to get container status \"dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc\": rpc error: code = NotFound desc = could not find container \"dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc\": container with ID starting with dae348fb40c3425b917b62cf7f3ddbefb06c146a4ec78fb18d1a0fb6823c92cc not found: ID does not exist" Nov 02 08:38:11.555256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.555202 5055 scope.go:122] "RemoveContainer" containerID="e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1" Nov 02 08:38:11.555546 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:11.555529 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1\": container with ID starting with e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1 not found: ID does not exist" containerID="e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1" Nov 02 08:38:11.555603 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.555550 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1"} err="failed to get container status \"e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1\": rpc error: code = NotFound desc = could not find container \"e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1\": container with ID starting with e1da9f211825eb981c12fafad08f6140eaf6afb0acdb432188b3c2b385bae4d1 not found: ID does not exist" Nov 02 08:38:11.630189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.629569 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podStartSLOduration=1.6295465359999999 podStartE2EDuration="1.629546536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:11.581843122 +0000 UTC m=+1938.672463543" watchObservedRunningTime="2026-11-02 08:38:11.629546536 +0000 UTC m=+1938.720166957" Nov 02 08:38:11.730551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.730494 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd"] Nov 02 08:38:11.736236 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.736205 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c17125-openshift-default-58cf794799-q4xqd"] Nov 02 08:38:11.827468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.827373 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b"] Nov 02 08:38:11.834653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.834629 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:11.926237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:11.926197 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc"] Nov 02 08:38:12.836653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:12.836616 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 08:38:12.837159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:12.836673 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 08:38:13.145258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:13.145183 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be57046b-d6ac-417b-b856-2a940616bd25" path="/var/lib/kubelet/pods/be57046b-d6ac-417b-b856-2a940616bd25/volumes" Nov 02 08:38:13.858452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:13.848625 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 08:38:13.858452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:13.848728 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 08:38:14.375165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:14.374111 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 08:38:14.835167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:14.834689 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 08:38:14.835167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:14.834767 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 08:38:15.835367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:15.835270 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 08:38:15.835367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:15.835339 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 08:38:16.831154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:16.831101 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" podUID="92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" containerName="payload-processing" containerID="cri-o://393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890" gracePeriod=25 Nov 02 08:38:16.834860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:16.834826 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 08:38:16.835013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:16.834882 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 08:38:16.927750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:16.927700 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" podUID="8649dacf-8185-4422-9869-872b3b545dc8" containerName="payload-pre-processing" containerID="cri-o://55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7" gracePeriod=25 Nov 02 08:38:17.052110 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:17.052080 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8649dacf_8185_4422_9869_872b3b545dc8.slice/crio-conmon-55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7.scope\": RecentStats: unable to find data in memory cache]" Nov 02 08:38:17.325336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.324877 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:38:17.418939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.418912 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:38:17.427402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.427369 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume\") pod \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " Nov 02 08:38:17.427656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.427632 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf\" (UniqueName: \"kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf\") pod \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\" (UID: \"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0\") " Nov 02 08:38:17.427845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.427815 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume" pod "92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" (UID: "92b386f8-6a6e-40e9-909b-4fb8b7da6ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:17.428077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.428056 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:17.429828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.429801 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf" pod "92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" (UID: "92b386f8-6a6e-40e9-909b-4fb8b7da6ad0"). InnerVolumeSpecName "kube-api-access-95prf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:17.529638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.529603 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume\") pod \"8649dacf-8185-4422-9869-872b3b545dc8\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " Nov 02 08:38:17.529844 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.529760 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd\" (UniqueName: \"kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd\") pod \"8649dacf-8185-4422-9869-872b3b545dc8\" (UID: \"8649dacf-8185-4422-9869-872b3b545dc8\") " Nov 02 08:38:17.530092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.530050 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume" pod "8649dacf-8185-4422-9869-872b3b545dc8" (UID: "8649dacf-8185-4422-9869-872b3b545dc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:17.530243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.530133 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95prf\" (UniqueName: \"kubernetes.io/projected/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0-kube-api-access-95prf\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:17.532509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.532478 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd" pod "8649dacf-8185-4422-9869-872b3b545dc8" (UID: "8649dacf-8185-4422-9869-872b3b545dc8"). InnerVolumeSpecName "kube-api-access-frptd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:17.631679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.631633 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8649dacf-8185-4422-9869-872b3b545dc8-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:17.631679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.631673 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frptd\" (UniqueName: \"kubernetes.io/projected/8649dacf-8185-4422-9869-872b3b545dc8-kube-api-access-frptd\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:17.675402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.675316 5055 generic.go:356] "Generic (PLEG): container finished" podID="92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" containerID="393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890" exitCode=0 Nov 02 08:38:17.675402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.675368 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" Nov 02 08:38:17.675614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.675403 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" event={"ID":"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0","Type":"ContainerDied","Data":"393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890"} Nov 02 08:38:17.675614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.675487 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b" event={"ID":"92b386f8-6a6e-40e9-909b-4fb8b7da6ad0","Type":"ContainerDied","Data":"a4739e66d4b52c249be4421fa431b38653988ee363b8eb081fb3db06b0825b06"} Nov 02 08:38:17.675614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.675517 5055 scope.go:122] "RemoveContainer" containerID="393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890" Nov 02 08:38:17.677115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.677090 5055 generic.go:356] "Generic (PLEG): container finished" podID="8649dacf-8185-4422-9869-872b3b545dc8" containerID="55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7" exitCode=0 Nov 02 08:38:17.677285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.677125 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" event={"ID":"8649dacf-8185-4422-9869-872b3b545dc8","Type":"ContainerDied","Data":"55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7"} Nov 02 08:38:17.677285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.677183 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" Nov 02 08:38:17.677285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.677191 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc" event={"ID":"8649dacf-8185-4422-9869-872b3b545dc8","Type":"ContainerDied","Data":"cface5bda18ede6cb5e66db1feeb3e4ba6e0d31fcf247a27f4a38382d9a770c6"} Nov 02 08:38:17.716357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.716330 5055 scope.go:122] "RemoveContainer" containerID="393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890" Nov 02 08:38:17.716821 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:17.716787 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890\": container with ID starting with 393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890 not found: ID does not exist" containerID="393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890" Nov 02 08:38:17.716918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.716833 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890"} err="failed to get container status \"393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890\": rpc error: code = NotFound desc = could not find container \"393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890\": container with ID starting with 393150b1b0609ab96c5a2da7e031fdf19137a40c4a599611ee973708ee71a890 not found: ID does not exist" Nov 02 08:38:17.716918 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.716860 5055 scope.go:122] "RemoveContainer" containerID="55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7" Nov 02 08:38:17.729094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.729075 5055 scope.go:122] "RemoveContainer" containerID="55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7" Nov 02 08:38:17.729449 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:17.729425 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7\": container with ID starting with 55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7 not found: ID does not exist" containerID="55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7" Nov 02 08:38:17.729530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.729459 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7"} err="failed to get container status \"55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7\": rpc error: code = NotFound desc = could not find container \"55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7\": container with ID starting with 55fb84fdeccd1fe0bbdc5409947e7b18d11f71aed3d3bc5d79418d6641b076d7 not found: ID does not exist" Nov 02 08:38:17.832670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.832633 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b"] Nov 02 08:38:17.834366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.834296 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 08:38:17.834366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.834350 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 08:38:17.837477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.837449 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-7r62b"] Nov 02 08:38:17.940290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.940199 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc"] Nov 02 08:38:17.944321 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:17.944281 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cba6f9af-7c67c7fc87-nnltc"] Nov 02 08:38:18.983985 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:18.983856 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:18.986220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:18.985501 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:18.994522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:18.994452 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:38:19.102856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:19.102794 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8649dacf-8185-4422-9869-872b3b545dc8" path="/var/lib/kubelet/pods/8649dacf-8185-4422-9869-872b3b545dc8/volumes" Nov 02 08:38:19.103896 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:19.103849 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" path="/var/lib/kubelet/pods/92b386f8-6a6e-40e9-909b-4fb8b7da6ad0/volumes" Nov 02 08:38:20.708799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:20.708765 5055 generic.go:356] "Generic (PLEG): container finished" podID="108ddb47-d760-4bc5-97cc-25716febd86a" containerID="ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36" exitCode=0 Nov 02 08:38:20.709299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:20.708839 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" event={"ID":"108ddb47-d760-4bc5-97cc-25716febd86a","Type":"ContainerDied","Data":"ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36"} Nov 02 08:38:20.709402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:20.709383 5055 scope.go:122] "RemoveContainer" containerID="ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36" Nov 02 08:38:20.710883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:20.710847 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:20.710944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:20.710917 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:20.710980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:20.710951 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:21.721470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:21.721429 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" event={"ID":"108ddb47-d760-4bc5-97cc-25716febd86a","Type":"ContainerStarted","Data":"df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618"} Nov 02 08:38:22.100228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:22.093554 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:38:22.105363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:22.105305 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 08:38:22.105563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:22.105379 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 08:38:23.113038 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:23.110678 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 08:38:23.113038 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:23.110739 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 08:38:24.090169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:24.089716 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 08:38:24.090169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:24.089839 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 08:38:24.943516 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:24.943474 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:38:25.100175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.097309 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 08:38:25.100175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.097410 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 08:38:25.306165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.298624 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:38:25.628958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.628865 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7"] Nov 02 08:38:25.629635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629609 5055 cpu_manager.go:414] "containerMap: removing container" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" Nov 02 08:38:25.629635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629631 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" Nov 02 08:38:25.629822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629655 5055 cpu_manager.go:414] "containerMap: removing container" podUID="92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" containerName="payload-processing" Nov 02 08:38:25.629822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629665 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" containerName="payload-processing" Nov 02 08:38:25.629822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629676 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8649dacf-8185-4422-9869-872b3b545dc8" containerName="payload-pre-processing" Nov 02 08:38:25.629822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629683 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8649dacf-8185-4422-9869-872b3b545dc8" containerName="payload-pre-processing" Nov 02 08:38:25.629961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629840 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" Nov 02 08:38:25.629961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629856 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" Nov 02 08:38:25.629961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629862 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b386f8-6a6e-40e9-909b-4fb8b7da6ad0" containerName="payload-processing" Nov 02 08:38:25.629961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.629869 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8649dacf-8185-4422-9869-872b3b545dc8" containerName="payload-pre-processing" Nov 02 08:38:25.632720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.632701 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.724194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.724153 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7"] Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730410 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730476 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730536 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730582 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730713 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730737 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730792 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.731040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.730822 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832298 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832351 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832437 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwms\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832473 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832605 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832649 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832708 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.832834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.832773 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.834591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.833453 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.834591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.833615 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.834591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.833634 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.834591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.833773 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.834591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.833985 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.836207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.835879 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.836333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.836223 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:25.934095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:25.933998 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwms\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:26.036489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.036457 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:26.036973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.036605 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwms\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:26.102164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.099660 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 08:38:26.102164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.099724 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 08:38:26.247913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.247872 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:26.962647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.962071 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7"] Nov 02 08:38:26.996378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.996328 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:26.996501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.996435 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:26.996501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:26.996480 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:38:27.129965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.129927 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 08:38:27.130550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.130513 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 08:38:27.318251 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.317751 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5"] Nov 02 08:38:27.318847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.318817 5055 cpu_manager.go:414] "containerMap: removing container" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" Nov 02 08:38:27.318847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.318847 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be57046b-d6ac-417b-b856-2a940616bd25" containerName="istio-proxy" Nov 02 08:38:27.322814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.322785 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5"] Nov 02 08:38:27.322950 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.322929 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.386161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.382109 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f4516e-dockercfg-p7k4s\"" Nov 02 08:38:27.386520 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.386496 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f4516e\"" Nov 02 08:38:27.409639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.408974 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smg28\" (UniqueName: \"kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28\") pod \"payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.409639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.409182 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.511409 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.511306 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg28\" (UniqueName: \"kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28\") pod \"payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.511582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.511488 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.512614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.512585 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.543588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.543546 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl"] Nov 02 08:38:27.547531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.547496 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00cc0e-jfzcl"] Nov 02 08:38:27.565615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.565583 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg28\" (UniqueName: \"kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28\") pod \"payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.670011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.669974 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:27.954607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.954574 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" event={"ID":"5744df3f-50af-4c01-9427-c7083bac2056","Type":"ContainerStarted","Data":"a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb"} Nov 02 08:38:27.954774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:27.954629 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" event={"ID":"5744df3f-50af-4c01-9427-c7083bac2056","Type":"ContainerStarted","Data":"2064c036ae85ca02fca9b4cfdf5780b1eeaa30b79e0e907e9a48bc5bd212a90d"} Nov 02 08:38:28.045061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.045030 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5"] Nov 02 08:38:28.045480 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:38:28.045449 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57559d64_6360_4b38_9ad7_0edde9d7ed18.slice/crio-a1192b5a265994abccbc1e1aa3c388ad176d8823bdfbf1d07601c753dcadd742 WatchSource:0}: Error finding container a1192b5a265994abccbc1e1aa3c388ad176d8823bdfbf1d07601c753dcadd742: Status 404 returned error can't find the container with id a1192b5a265994abccbc1e1aa3c388ad176d8823bdfbf1d07601c753dcadd742 Nov 02 08:38:28.064979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.064923 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podStartSLOduration=3.064907527 podStartE2EDuration="3.064907527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:28.06266876 +0000 UTC m=+1955.153289179" watchObservedRunningTime="2026-11-02 08:38:28.064907527 +0000 UTC m=+1955.155527945" Nov 02 08:38:28.092386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.092348 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:38:28.093755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.093731 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:38:28.248800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.248707 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:28.250199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.250168 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 08:38:28.250342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.250227 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 08:38:28.966214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.966168 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" event={"ID":"57559d64-6360-4b38-9ad7-0edde9d7ed18","Type":"ContainerStarted","Data":"2fe1fda5f04ad73616bf5d93f6c21432ebf2d9cc5afd1deab7d6bb98bdf697a8"} Nov 02 08:38:28.966403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.966242 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" event={"ID":"57559d64-6360-4b38-9ad7-0edde9d7ed18","Type":"ContainerStarted","Data":"a1192b5a265994abccbc1e1aa3c388ad176d8823bdfbf1d07601c753dcadd742"} Nov 02 08:38:28.966403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:28.966296 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:29.075340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:29.075271 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" podStartSLOduration=2.075251009 podStartE2EDuration="2.075251009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:29.073548679 +0000 UTC m=+1956.164169098" watchObservedRunningTime="2026-11-02 08:38:29.075251009 +0000 UTC m=+1956.165871428" Nov 02 08:38:29.119095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:29.119035 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f801fc3-bad7-44eb-b816-aa912fb745c1" path="/var/lib/kubelet/pods/3f801fc3-bad7-44eb-b816-aa912fb745c1/volumes" Nov 02 08:38:29.255220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:29.249413 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 08:38:29.255220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:29.249479 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 08:38:30.248664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:30.248619 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 08:38:30.248831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:30.248686 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 08:38:31.046346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:31.046308 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb"] Nov 02 08:38:31.121055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:31.121020 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce12f9b-dlqbb"] Nov 02 08:38:31.249181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:31.249127 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 08:38:31.249352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:31.249208 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 08:38:32.248513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:32.248476 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 08:38:32.248920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:32.248527 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 08:38:33.123887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:33.123850 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e40792-3b31-49e5-8440-33e6250e99c6" path="/var/lib/kubelet/pods/76e40792-3b31-49e5-8440-33e6250e99c6/volumes" Nov 02 08:38:33.248943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:33.248906 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 08:38:33.249354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:33.248972 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 08:38:34.332195 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:34.332159 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4"] Nov 02 08:38:34.333366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:34.333328 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" containerID="cri-o://20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc" gracePeriod=30 Nov 02 08:38:34.623924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:34.623820 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:34.624630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:34.624605 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:34.625959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:34.625907 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:38:38.394294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:38.394254 5055 patch_prober.go:29] interesting pod/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:38:38.394728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:38.394338 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:38:39.667095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.667062 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:38:39.783081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783042 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783155 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783210 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783241 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783323 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783362 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783440 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783554 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:39.783661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783563 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783702 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy\") pod \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\" (UID: \"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a\") " Nov 02 08:38:39.783767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.783725 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:39.784780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.784026 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:39.784780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.784462 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.784780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.784486 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.784780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.784500 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.784780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.784567 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:39.784780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.784737 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:39.786378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.786227 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:38:39.786508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.786439 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:39.786990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.786693 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:39.786990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.786780 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h" pod "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" (UID: "9c29d016-c9d2-43c7-b20f-d43a58cd8a5a"). InnerVolumeSpecName "kube-api-access-b254h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:39.886095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.885991 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b254h\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-kube-api-access-b254h\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.886095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.886027 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.886095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.886038 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.886095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.886052 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.886095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.886061 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.886095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.886072 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:39.977878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:39.977846 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:38:40.219185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.219130 5055 generic.go:356] "Generic (PLEG): container finished" podID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerID="20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc" exitCode=0 Nov 02 08:38:40.219382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.219199 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" Nov 02 08:38:40.219382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.219213 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" event={"ID":"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a","Type":"ContainerDied","Data":"20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc"} Nov 02 08:38:40.219382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.219290 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4" event={"ID":"9c29d016-c9d2-43c7-b20f-d43a58cd8a5a","Type":"ContainerDied","Data":"6bd119c9f4794d97ecabc0392ab329ed22ebea4defbd5d48610dc699148eeb66"} Nov 02 08:38:40.219382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.219321 5055 scope.go:122] "RemoveContainer" containerID="20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc" Nov 02 08:38:40.233902 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.233874 5055 scope.go:122] "RemoveContainer" containerID="42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2" Nov 02 08:38:40.244580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.244550 5055 scope.go:122] "RemoveContainer" containerID="20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc" Nov 02 08:38:40.244880 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:40.244859 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc\": container with ID starting with 20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc not found: ID does not exist" containerID="20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc" Nov 02 08:38:40.244948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.244889 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc"} err="failed to get container status \"20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc\": rpc error: code = NotFound desc = could not find container \"20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc\": container with ID starting with 20545ab3f52386fc52dd61be252170e2867162029dccd29297c378ff2a89c9cc not found: ID does not exist" Nov 02 08:38:40.244948 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.244909 5055 scope.go:122] "RemoveContainer" containerID="42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2" Nov 02 08:38:40.245161 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:40.245125 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2\": container with ID starting with 42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2 not found: ID does not exist" containerID="42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2" Nov 02 08:38:40.245254 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.245166 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2"} err="failed to get container status \"42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2\": rpc error: code = NotFound desc = could not find container \"42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2\": container with ID starting with 42e30e1f658dc34311df6badd8821760314d9560b6651fefb2e8d41f92fdc8c2 not found: ID does not exist" Nov 02 08:38:40.376018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.375968 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4"] Nov 02 08:38:40.379650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:40.379624 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-00cc0e-openshift-default-748c64f44c-28sg4"] Nov 02 08:38:41.105212 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:41.105173 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" path="/var/lib/kubelet/pods/9c29d016-c9d2-43c7-b20f-d43a58cd8a5a/volumes" Nov 02 08:38:45.072366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.072318 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx"] Nov 02 08:38:45.073208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.073179 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" Nov 02 08:38:45.073208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.073205 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" Nov 02 08:38:45.073396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.073239 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" Nov 02 08:38:45.073396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.073247 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" Nov 02 08:38:45.073493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.073434 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" Nov 02 08:38:45.073493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.073458 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c29d016-c9d2-43c7-b20f-d43a58cd8a5a" containerName="istio-proxy" Nov 02 08:38:45.077476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.077456 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.126165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.126116 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:38:45.126373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.126123 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:38:45.164828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.164775 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xb2r\" (UniqueName: \"kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.165112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.164865 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.165112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.164892 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.172022 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.171971 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:38:45.172202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.172113 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:38:45.180559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.179288 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx"] Nov 02 08:38:45.267072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.267026 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xb2r\" (UniqueName: \"kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.267256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.267083 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.267256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.267111 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.267960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.267932 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.269861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.269833 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.322508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.322434 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xb2r\" (UniqueName: \"kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r\") pod \"maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.393155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.393111 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:45.774393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:45.774361 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx"] Nov 02 08:38:45.775033 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:38:45.775006 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74175dd_1bbc_4800_a41f_a8a5b2d74fc4.slice/crio-e9ed5c35c8f7abffb9791501cfe3a6353d5e73b6bce21bd12866d7e31d890f5c WatchSource:0}: Error finding container e9ed5c35c8f7abffb9791501cfe3a6353d5e73b6bce21bd12866d7e31d890f5c: Status 404 returned error can't find the container with id e9ed5c35c8f7abffb9791501cfe3a6353d5e73b6bce21bd12866d7e31d890f5c Nov 02 08:38:46.279291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:46.279249 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" event={"ID":"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4","Type":"ContainerStarted","Data":"e9ed5c35c8f7abffb9791501cfe3a6353d5e73b6bce21bd12866d7e31d890f5c"} Nov 02 08:38:47.289550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:47.289511 5055 generic.go:356] "Generic (PLEG): container finished" podID="b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" containerID="467c53126b5404a6acaa408bfbd03dc43ac62773085b4edc7196a49da258075b" exitCode=0 Nov 02 08:38:47.289947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:47.289595 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" event={"ID":"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4","Type":"ContainerDied","Data":"467c53126b5404a6acaa408bfbd03dc43ac62773085b4edc7196a49da258075b"} Nov 02 08:38:48.494852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.494816 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:48.638468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.638369 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca\") pod \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " Nov 02 08:38:48.638468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.638435 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r\" (UniqueName: \"kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r\") pod \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " Nov 02 08:38:48.638774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.638751 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca\") pod \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\" (UID: \"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4\") " Nov 02 08:38:48.638847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.638823 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca" pod "b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" (UID: "b74175dd-1bbc-4800-a41f-a8a5b2d74fc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:48.639443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.639418 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:48.641247 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.641216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca" pod "b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" (UID: "b74175dd-1bbc-4800-a41f-a8a5b2d74fc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:48.641740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.641713 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r" pod "b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" (UID: "b74175dd-1bbc-4800-a41f-a8a5b2d74fc4"). InnerVolumeSpecName "kube-api-access-9xb2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:48.750586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.745602 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:48.750586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:48.750514 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xb2r\" (UniqueName: \"kubernetes.io/projected/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4-kube-api-access-9xb2r\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:49.313008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:49.312974 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" Nov 02 08:38:49.313265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:49.312975 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx" event={"ID":"b74175dd-1bbc-4800-a41f-a8a5b2d74fc4","Type":"ContainerDied","Data":"e9ed5c35c8f7abffb9791501cfe3a6353d5e73b6bce21bd12866d7e31d890f5c"} Nov 02 08:38:49.313265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:49.313108 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ed5c35c8f7abffb9791501cfe3a6353d5e73b6bce21bd12866d7e31d890f5c" Nov 02 08:38:54.926382 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:38:54.926341 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:38:54.995798 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:54.995757 5055 scope.go:122] "RemoveContainer" containerID="b32b2c8c61bd1cb0c026ee2e4fae0360d7b027d9dcaf6d8fa9ebdbef2f73084d" Nov 02 08:38:56.292992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:56.292957 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb"] Nov 02 08:38:56.294260 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:56.294221 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" containerID="cri-o://ecf82e517c9202998106d696502321f380e15fe900bf258dbb06bcfb9143bd27" gracePeriod=30 Nov 02 08:38:58.435351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:58.435307 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:38:58.435792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:58.435384 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:38:59.934945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:38:59.934901 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:39:01.094297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.094260 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt"] Nov 02 08:39:01.156390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.156353 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cba6f9af-s9lnt"] Nov 02 08:39:01.541105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.541068 5055 generic.go:356] "Generic (PLEG): container finished" podID="a53e4327-185b-4482-860c-784fc93cc695" containerID="ecf82e517c9202998106d696502321f380e15fe900bf258dbb06bcfb9143bd27" exitCode=0 Nov 02 08:39:01.545163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.541468 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" event={"ID":"a53e4327-185b-4482-860c-784fc93cc695","Type":"ContainerDied","Data":"ecf82e517c9202998106d696502321f380e15fe900bf258dbb06bcfb9143bd27"} Nov 02 08:39:01.864336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.864300 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:39:01.936874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.936839 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.936897 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.936934 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.936992 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.937105 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.937134 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.937172 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:01.937284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.937237 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.937294 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.937512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.937354 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w\") pod \"a53e4327-185b-4482-860c-784fc93cc695\" (UID: \"a53e4327-185b-4482-860c-784fc93cc695\") " Nov 02 08:39:01.938198 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.938164 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:01.938396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.938166 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:01.938396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.938167 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:01.938396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.938213 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:01.938396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.938377 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:01.939538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.939512 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:01.939783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.939757 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:01.939946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.939925 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:01.940235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:01.940212 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w" pod "a53e4327-185b-4482-860c-784fc93cc695" (UID: "a53e4327-185b-4482-860c-784fc93cc695"). InnerVolumeSpecName "kube-api-access-2cz7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:02.040080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.039988 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53e4327-185b-4482-860c-784fc93cc695-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040031 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040046 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53e4327-185b-4482-860c-784fc93cc695-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040060 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040080 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040073 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040086 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cz7w\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-kube-api-access-2cz7w\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040102 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53e4327-185b-4482-860c-784fc93cc695-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.040397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.040115 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53e4327-185b-4482-860c-784fc93cc695-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:02.551981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.551946 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" Nov 02 08:39:02.552479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.551969 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb" event={"ID":"a53e4327-185b-4482-860c-784fc93cc695","Type":"ContainerDied","Data":"e2c11556166e453e46be94480ab02844bd5e6aedb87a2bd7d7db22a113e3f3b9"} Nov 02 08:39:02.552479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.552066 5055 scope.go:122] "RemoveContainer" containerID="ecf82e517c9202998106d696502321f380e15fe900bf258dbb06bcfb9143bd27" Nov 02 08:39:02.753797 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.753751 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb"] Nov 02 08:39:02.799419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:02.799382 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-x5gwb"] Nov 02 08:39:03.094856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:03.094823 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da564fc-50a5-46f2-ada7-e33ffe2f8d43" path="/var/lib/kubelet/pods/5da564fc-50a5-46f2-ada7-e33ffe2f8d43/volumes" Nov 02 08:39:03.095313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:03.095299 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53e4327-185b-4482-860c-784fc93cc695" path="/var/lib/kubelet/pods/a53e4327-185b-4482-860c-784fc93cc695/volumes" Nov 02 08:39:05.106470 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:05.106430 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l"] Nov 02 08:39:05.208661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:05.208613 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn"] Nov 02 08:39:07.445153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:07.445109 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78"] Nov 02 08:39:07.491111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:07.491069 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6a289-9lq78"] Nov 02 08:39:09.097112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:09.097068 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bef17e4-39d0-4528-99e8-64eff2f5d710" path="/var/lib/kubelet/pods/3bef17e4-39d0-4528-99e8-64eff2f5d710/volumes" Nov 02 08:39:10.107204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.107136 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" podUID="9c62f0b5-26f5-4f5d-ba28-1e4966374af6" containerName="payload-processing" containerID="cri-o://18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572" gracePeriod=25 Nov 02 08:39:10.209192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.209109 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" podUID="22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" containerName="payload-pre-processing" containerID="cri-o://26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650" gracePeriod=25 Nov 02 08:39:10.423443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.423410 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:39:10.486668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.486631 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4\" (UniqueName: \"kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4\") pod \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " Nov 02 08:39:10.486869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.486842 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume\") pod \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\" (UID: \"9c62f0b5-26f5-4f5d-ba28-1e4966374af6\") " Nov 02 08:39:10.487246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.487220 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume" pod "9c62f0b5-26f5-4f5d-ba28-1e4966374af6" (UID: "9c62f0b5-26f5-4f5d-ba28-1e4966374af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:10.487756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.487735 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:10.488781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.488758 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4" pod "9c62f0b5-26f5-4f5d-ba28-1e4966374af6" (UID: "9c62f0b5-26f5-4f5d-ba28-1e4966374af6"). InnerVolumeSpecName "kube-api-access-lrwf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:10.505281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.505252 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:39:10.588816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.588782 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9\" (UniqueName: \"kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9\") pod \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " Nov 02 08:39:10.589017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.588946 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume\") pod \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\" (UID: \"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1\") " Nov 02 08:39:10.589431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.589398 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume" pod "22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" (UID: "22056cb3-e9d3-4c56-9a3f-b08ab1196ab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:10.589880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.589857 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrwf4\" (UniqueName: \"kubernetes.io/projected/9c62f0b5-26f5-4f5d-ba28-1e4966374af6-kube-api-access-lrwf4\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:10.589970 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.589885 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:10.591120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.591099 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9" pod "22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" (UID: "22056cb3-e9d3-4c56-9a3f-b08ab1196ab1"). InnerVolumeSpecName "kube-api-access-tdqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:10.632373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.632335 5055 generic.go:356] "Generic (PLEG): container finished" podID="22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" containerID="26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650" exitCode=0 Nov 02 08:39:10.632373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.632376 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" Nov 02 08:39:10.632605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.632425 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" event={"ID":"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1","Type":"ContainerDied","Data":"26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650"} Nov 02 08:39:10.632605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.632486 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn" event={"ID":"22056cb3-e9d3-4c56-9a3f-b08ab1196ab1","Type":"ContainerDied","Data":"65902549d1f46159fdef70ab20ad22470697985b5ca6d84c155469304932098a"} Nov 02 08:39:10.632605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.632515 5055 scope.go:122] "RemoveContainer" containerID="26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650" Nov 02 08:39:10.634048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.634018 5055 generic.go:356] "Generic (PLEG): container finished" podID="9c62f0b5-26f5-4f5d-ba28-1e4966374af6" containerID="18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572" exitCode=0 Nov 02 08:39:10.634257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.634070 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" Nov 02 08:39:10.634257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.634079 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" event={"ID":"9c62f0b5-26f5-4f5d-ba28-1e4966374af6","Type":"ContainerDied","Data":"18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572"} Nov 02 08:39:10.634257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.634116 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l" event={"ID":"9c62f0b5-26f5-4f5d-ba28-1e4966374af6","Type":"ContainerDied","Data":"b7f0ee019f201162b61efcc2ef1cf9db73f44c4f7f216037db05756d6f6d8e6a"} Nov 02 08:39:10.646574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.646547 5055 scope.go:122] "RemoveContainer" containerID="26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650" Nov 02 08:39:10.647068 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:10.647041 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650\": container with ID starting with 26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650 not found: ID does not exist" containerID="26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650" Nov 02 08:39:10.647131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.647078 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650"} err="failed to get container status \"26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650\": rpc error: code = NotFound desc = could not find container \"26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650\": container with ID starting with 26c6235be9f8b8ac68fd7e223139b02fc133a3ac7c654b42f9de48e89cbf7650 not found: ID does not exist" Nov 02 08:39:10.647131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.647096 5055 scope.go:122] "RemoveContainer" containerID="18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572" Nov 02 08:39:10.660595 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.660568 5055 scope.go:122] "RemoveContainer" containerID="18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572" Nov 02 08:39:10.662724 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:10.662692 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572\": container with ID starting with 18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572 not found: ID does not exist" containerID="18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572" Nov 02 08:39:10.662850 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.662730 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572"} err="failed to get container status \"18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572\": rpc error: code = NotFound desc = could not find container \"18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572\": container with ID starting with 18bf637ac458c032dd6c88b3818a522b985bbea5f1eb66735e8038052e195572 not found: ID does not exist" Nov 02 08:39:10.691312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.691273 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqx9\" (UniqueName: \"kubernetes.io/projected/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1-kube-api-access-tdqx9\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:10.840014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.839976 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn"] Nov 02 08:39:10.843699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.843666 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45fc31-67b6c6fc67-vkfxn"] Nov 02 08:39:10.942913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.942831 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l"] Nov 02 08:39:10.947280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:10.947251 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45fc31-7d5c46c999-9c87l"] Nov 02 08:39:11.093462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:11.093426 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" path="/var/lib/kubelet/pods/22056cb3-e9d3-4c56-9a3f-b08ab1196ab1/volumes" Nov 02 08:39:11.093933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:11.093919 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c62f0b5-26f5-4f5d-ba28-1e4966374af6" path="/var/lib/kubelet/pods/9c62f0b5-26f5-4f5d-ba28-1e4966374af6/volumes" Nov 02 08:39:13.994169 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:13.994124 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s"] Nov 02 08:39:14.040077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:14.040030 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7f7d7-mp48s"] Nov 02 08:39:15.096659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:15.096614 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b6e323-079f-4bde-801a-1470f6b74f13" path="/var/lib/kubelet/pods/b6b6e323-079f-4bde-801a-1470f6b74f13/volumes" Nov 02 08:39:16.432736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:16.432700 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp"] Nov 02 08:39:20.410136 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:20.410100 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv"] Nov 02 08:39:20.411801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:20.410374 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" containerID="cri-o://28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b" gracePeriod=30 Nov 02 08:39:24.419993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:24.419944 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm"] Nov 02 08:39:24.420841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:24.420804 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" containerID="cri-o://3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656" gracePeriod=30 Nov 02 08:39:24.923361 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:24.923318 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:39:25.398304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.398268 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2"] Nov 02 08:39:25.398955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.398936 5055 cpu_manager.go:414] "containerMap: removing container" podUID="22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" containerName="payload-pre-processing" Nov 02 08:39:25.398955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.398953 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" containerName="payload-pre-processing" Nov 02 08:39:25.399109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399000 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" containerName="revoke-keys" Nov 02 08:39:25.399109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399005 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" containerName="revoke-keys" Nov 02 08:39:25.399109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399015 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9c62f0b5-26f5-4f5d-ba28-1e4966374af6" containerName="payload-processing" Nov 02 08:39:25.399109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399020 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c62f0b5-26f5-4f5d-ba28-1e4966374af6" containerName="payload-processing" Nov 02 08:39:25.399109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399030 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" Nov 02 08:39:25.399109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399035 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" Nov 02 08:39:25.399346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399125 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c62f0b5-26f5-4f5d-ba28-1e4966374af6" containerName="payload-processing" Nov 02 08:39:25.399346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399150 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53e4327-185b-4482-860c-784fc93cc695" containerName="istio-proxy" Nov 02 08:39:25.399346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399184 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="22056cb3-e9d3-4c56-9a3f-b08ab1196ab1" containerName="payload-pre-processing" Nov 02 08:39:25.399346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.399200 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" containerName="revoke-keys" Nov 02 08:39:25.403190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.403166 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.454915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.454878 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.455311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.455069 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.455311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.455269 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnbf\" (UniqueName: \"kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.500833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.500801 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:39:25.501035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.500886 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:39:25.501035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.500985 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:39:25.501035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.500989 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:39:25.504986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.504960 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2"] Nov 02 08:39:25.556185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.556119 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.556400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.556235 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.556400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.556360 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnbf\" (UniqueName: \"kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.557006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.556963 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.562183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.559922 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.616189 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.615225 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnbf\" (UniqueName: \"kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf\") pod \"maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.701591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.701560 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:39:25.717407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.717369 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:25.759267 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759232 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759344 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759446 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759485 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759569 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759547 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759599 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759626 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:25.759675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759645 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759752 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759788 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs\") pod \"3fc7682d-cdbd-4d67-8b92-97849eb33547\" (UID: \"3fc7682d-cdbd-4d67-8b92-97849eb33547\") " Nov 02 08:39:25.759904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759876 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:25.759996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.759956 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:25.760355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.760327 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:25.760573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.760549 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.760695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.760577 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.760695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.760594 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.760695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.760609 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc7682d-cdbd-4d67-8b92-97849eb33547-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.761889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.761865 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:25.762188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.762120 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:25.762431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.762413 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:25.762493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.762438 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:25.764127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.763901 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq" pod "3fc7682d-cdbd-4d67-8b92-97849eb33547" (UID: "3fc7682d-cdbd-4d67-8b92-97849eb33547"). InnerVolumeSpecName "kube-api-access-58sdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:25.774201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.772868 5055 generic.go:356] "Generic (PLEG): container finished" podID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerID="28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b" exitCode=0 Nov 02 08:39:25.774201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.773015 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" Nov 02 08:39:25.774201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.773197 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" event={"ID":"3fc7682d-cdbd-4d67-8b92-97849eb33547","Type":"ContainerDied","Data":"28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b"} Nov 02 08:39:25.774201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.773243 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv" event={"ID":"3fc7682d-cdbd-4d67-8b92-97849eb33547","Type":"ContainerDied","Data":"3d4f075653a34e98d1687e1f8b1c603e371a8ee31e75e162823762992ee2bfd7"} Nov 02 08:39:25.774201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.773271 5055 scope.go:122] "RemoveContainer" containerID="28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b" Nov 02 08:39:25.792284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.792250 5055 scope.go:122] "RemoveContainer" containerID="416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171" Nov 02 08:39:25.808202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.808181 5055 scope.go:122] "RemoveContainer" containerID="28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b" Nov 02 08:39:25.808631 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:25.808589 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b\": container with ID starting with 28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b not found: ID does not exist" containerID="28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b" Nov 02 08:39:25.808719 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.808635 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b"} err="failed to get container status \"28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b\": rpc error: code = NotFound desc = could not find container \"28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b\": container with ID starting with 28ed552af51274173fb5c048ed22d92ce78ab0abd48ec2e73345e5b877fb8e0b not found: ID does not exist" Nov 02 08:39:25.808719 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.808663 5055 scope.go:122] "RemoveContainer" containerID="416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171" Nov 02 08:39:25.808986 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:25.808961 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171\": container with ID starting with 416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171 not found: ID does not exist" containerID="416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171" Nov 02 08:39:25.809060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.808997 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171"} err="failed to get container status \"416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171\": rpc error: code = NotFound desc = could not find container \"416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171\": container with ID starting with 416f2a616cf584809862c8a8f4a7b1da67af05509af7c0d412bc411f79f93171 not found: ID does not exist" Nov 02 08:39:25.861810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.861760 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.861810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.861799 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58sdq\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-kube-api-access-58sdq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.861810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.861815 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.862194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.861830 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.862194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.861858 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc7682d-cdbd-4d67-8b92-97849eb33547-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:25.926218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.926086 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv"] Nov 02 08:39:25.930622 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:25.930587 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c7f7d7-openshift-default-7fd48d8c5f-wvqhv"] Nov 02 08:39:26.090980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:26.090944 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2"] Nov 02 08:39:26.092176 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:39:26.092109 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cbfd47_5703_4091_9985_dd7e40a6d611.slice/crio-7f71c06dfa3e1401aee580c383d2b5472ef079f2f2e2de14c3c78155cb22ab70 WatchSource:0}: Error finding container 7f71c06dfa3e1401aee580c383d2b5472ef079f2f2e2de14c3c78155cb22ab70: Status 404 returned error can't find the container with id 7f71c06dfa3e1401aee580c383d2b5472ef079f2f2e2de14c3c78155cb22ab70 Nov 02 08:39:26.782814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:26.782773 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" event={"ID":"97cbfd47-5703-4091-9985-dd7e40a6d611","Type":"ContainerStarted","Data":"ad702550dfdf959db74c96a355202d2b88b2b21f1486ce7ca9c8fbb9c210e327"} Nov 02 08:39:26.783378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:26.782837 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" event={"ID":"97cbfd47-5703-4091-9985-dd7e40a6d611","Type":"ContainerStarted","Data":"7f71c06dfa3e1401aee580c383d2b5472ef079f2f2e2de14c3c78155cb22ab70"} Nov 02 08:39:26.891587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:26.891520 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" podStartSLOduration=1.891506052 podStartE2EDuration="1.891506052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:26.888227491 +0000 UTC m=+2013.978847910" watchObservedRunningTime="2026-11-02 08:39:26.891506052 +0000 UTC m=+2013.982126471" Nov 02 08:39:27.094465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:27.094371 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" path="/var/lib/kubelet/pods/3fc7682d-cdbd-4d67-8b92-97849eb33547/volumes" Nov 02 08:39:27.800228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:27.800184 5055 generic.go:356] "Generic (PLEG): container finished" podID="97cbfd47-5703-4091-9985-dd7e40a6d611" containerID="ad702550dfdf959db74c96a355202d2b88b2b21f1486ce7ca9c8fbb9c210e327" exitCode=0 Nov 02 08:39:27.800749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:27.800342 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" event={"ID":"97cbfd47-5703-4091-9985-dd7e40a6d611","Type":"ContainerDied","Data":"ad702550dfdf959db74c96a355202d2b88b2b21f1486ce7ca9c8fbb9c210e327"} Nov 02 08:39:27.881632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:27.881585 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:39:27.881820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:27.881664 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:39:29.006284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.006244 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:29.106494 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.106390 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca\") pod \"97cbfd47-5703-4091-9985-dd7e40a6d611\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " Nov 02 08:39:29.106665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.106592 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf\" (UniqueName: \"kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf\") pod \"97cbfd47-5703-4091-9985-dd7e40a6d611\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " Nov 02 08:39:29.106748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.106733 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca\") pod \"97cbfd47-5703-4091-9985-dd7e40a6d611\" (UID: \"97cbfd47-5703-4091-9985-dd7e40a6d611\") " Nov 02 08:39:29.106851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.106825 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca" pod "97cbfd47-5703-4091-9985-dd7e40a6d611" (UID: "97cbfd47-5703-4091-9985-dd7e40a6d611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:29.107451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.107421 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.109045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.109011 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca" pod "97cbfd47-5703-4091-9985-dd7e40a6d611" (UID: "97cbfd47-5703-4091-9985-dd7e40a6d611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:29.109226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.109115 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf" pod "97cbfd47-5703-4091-9985-dd7e40a6d611" (UID: "97cbfd47-5703-4091-9985-dd7e40a6d611"). InnerVolumeSpecName "kube-api-access-clnbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:29.209576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.209478 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cbfd47-5703-4091-9985-dd7e40a6d611-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.209576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.209536 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnbf\" (UniqueName: \"kubernetes.io/projected/97cbfd47-5703-4091-9985-dd7e40a6d611-kube-api-access-clnbf\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.700544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.700509 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:39:29.822876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.822837 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.822995 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823039 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823157 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823260 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823339 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823424 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823468 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823466 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:29.823502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823482 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:29.823713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823593 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7\") pod \"048aa870-d17c-47ee-9ed0-583b3324f9d8\" (UID: \"048aa870-d17c-47ee-9ed0-583b3324f9d8\") " Nov 02 08:39:29.823807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823786 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:29.823878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.823835 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:29.824215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.824192 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:29.824894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.824868 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048aa870-d17c-47ee-9ed0-583b3324f9d8-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.825197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825178 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.825349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825327 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" Nov 02 08:39:29.825495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825336 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.825601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825587 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.825696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825677 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.825696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825626 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" event={"ID":"048aa870-d17c-47ee-9ed0-583b3324f9d8","Type":"ContainerDied","Data":"3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656"} Nov 02 08:39:29.825898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825762 5055 scope.go:122] "RemoveContainer" containerID="3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656" Nov 02 08:39:29.825898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.825575 5055 generic.go:356] "Generic (PLEG): container finished" podID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerID="3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656" exitCode=0 Nov 02 08:39:29.826161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.826084 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:29.826161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.826079 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm" event={"ID":"048aa870-d17c-47ee-9ed0-583b3324f9d8","Type":"ContainerDied","Data":"d7e2fc15669131b09dc6052c09e313449c1305b97f890895ef5493d343feb770"} Nov 02 08:39:29.826722 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.826699 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:29.827255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.827048 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "kube-api-access-2ldf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:29.827255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.827215 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy" pod "048aa870-d17c-47ee-9ed0-583b3324f9d8" (UID: "048aa870-d17c-47ee-9ed0-583b3324f9d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:29.829302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.829269 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" event={"ID":"97cbfd47-5703-4091-9985-dd7e40a6d611","Type":"ContainerDied","Data":"7f71c06dfa3e1401aee580c383d2b5472ef079f2f2e2de14c3c78155cb22ab70"} Nov 02 08:39:29.829423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.829323 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f71c06dfa3e1401aee580c383d2b5472ef079f2f2e2de14c3c78155cb22ab70" Nov 02 08:39:29.829423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.829335 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2" Nov 02 08:39:29.840366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.840337 5055 scope.go:122] "RemoveContainer" containerID="5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b" Nov 02 08:39:29.855202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.855115 5055 scope.go:122] "RemoveContainer" containerID="3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656" Nov 02 08:39:29.855541 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:29.855518 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656\": container with ID starting with 3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656 not found: ID does not exist" containerID="3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656" Nov 02 08:39:29.855593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.855551 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656"} err="failed to get container status \"3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656\": rpc error: code = NotFound desc = could not find container \"3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656\": container with ID starting with 3f5aaa69a78a88192a921a9e9725eb33eec603a6ee2dac30b0a9647d97685656 not found: ID does not exist" Nov 02 08:39:29.855593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.855575 5055 scope.go:122] "RemoveContainer" containerID="5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b" Nov 02 08:39:29.855856 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:29.855836 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b\": container with ID starting with 5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b not found: ID does not exist" containerID="5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b" Nov 02 08:39:29.855914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.855861 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b"} err="failed to get container status \"5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b\": rpc error: code = NotFound desc = could not find container \"5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b\": container with ID starting with 5ac218d86cee2f0938ec4ac1d8670a514679521d50a37464f5f68604c653c50b not found: ID does not exist" Nov 02 08:39:29.927062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.927010 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldf7\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-kube-api-access-2ldf7\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.927062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.927057 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.927062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.927074 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:29.927447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:29.927088 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048aa870-d17c-47ee-9ed0-583b3324f9d8-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:30.248105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:30.248014 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm"] Nov 02 08:39:30.255464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:30.255426 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6a289-openshift-default-5cfd69676f-ms5mm"] Nov 02 08:39:31.106855 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.106808 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" path="/var/lib/kubelet/pods/048aa870-d17c-47ee-9ed0-583b3324f9d8/volumes" Nov 02 08:39:31.457280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.457241 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" containerID="cri-o://b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c" gracePeriod=45 Nov 02 08:39:31.577652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.577614 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": dial tcp 10.132.0.124:8000: connect: connection refused" Nov 02 08:39:31.757324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.757290 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:39:31.846677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.846636 5055 generic.go:356] "Generic (PLEG): container finished" podID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerID="b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c" exitCode=0 Nov 02 08:39:31.846870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.846686 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" Nov 02 08:39:31.846870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.846719 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" event={"ID":"c6af0b3b-1a28-40f8-ac81-7fe5a440f968","Type":"ContainerDied","Data":"b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c"} Nov 02 08:39:31.846870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.846773 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp" event={"ID":"c6af0b3b-1a28-40f8-ac81-7fe5a440f968","Type":"ContainerDied","Data":"020ae4ef3dd9da17a050fdf5b655493c3c543ed74d61c5bb0b9ed6411f266658"} Nov 02 08:39:31.846870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.846793 5055 scope.go:122] "RemoveContainer" containerID="b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c" Nov 02 08:39:31.852274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852242 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs\") pod \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " Nov 02 08:39:31.852386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852283 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir\") pod \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " Nov 02 08:39:31.852471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852452 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache\") pod \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " Nov 02 08:39:31.852580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852556 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p\" (UniqueName: \"kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p\") pod \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " Nov 02 08:39:31.852580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852565 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir" pod "c6af0b3b-1a28-40f8-ac81-7fe5a440f968" (UID: "c6af0b3b-1a28-40f8-ac81-7fe5a440f968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:31.852783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852621 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home\") pod \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " Nov 02 08:39:31.852783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852659 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm\") pod \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\" (UID: \"c6af0b3b-1a28-40f8-ac81-7fe5a440f968\") " Nov 02 08:39:31.852783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852676 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache" pod "c6af0b3b-1a28-40f8-ac81-7fe5a440f968" (UID: "c6af0b3b-1a28-40f8-ac81-7fe5a440f968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:31.852943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.852837 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home" pod "c6af0b3b-1a28-40f8-ac81-7fe5a440f968" (UID: "c6af0b3b-1a28-40f8-ac81-7fe5a440f968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:31.853324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.853302 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:31.853324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.853324 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:31.853479 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.853333 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:31.854848 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.854824 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p" pod "c6af0b3b-1a28-40f8-ac81-7fe5a440f968" (UID: "c6af0b3b-1a28-40f8-ac81-7fe5a440f968"). InnerVolumeSpecName "kube-api-access-cr84p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:31.854961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.854894 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm" pod "c6af0b3b-1a28-40f8-ac81-7fe5a440f968" (UID: "c6af0b3b-1a28-40f8-ac81-7fe5a440f968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:31.854961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.854932 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs" pod "c6af0b3b-1a28-40f8-ac81-7fe5a440f968" (UID: "c6af0b3b-1a28-40f8-ac81-7fe5a440f968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:31.858839 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.858819 5055 scope.go:122] "RemoveContainer" containerID="b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c" Nov 02 08:39:31.859185 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:31.859152 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c\": container with ID starting with b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c not found: ID does not exist" containerID="b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c" Nov 02 08:39:31.859278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.859193 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c"} err="failed to get container status \"b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c\": rpc error: code = NotFound desc = could not find container \"b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c\": container with ID starting with b0cfb5aa62814d465969331559c15d07ad5a74813622cea91974dfa5418bfd3c not found: ID does not exist" Nov 02 08:39:31.919408 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.919374 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl"] Nov 02 08:39:31.920115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920091 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" Nov 02 08:39:31.920115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920115 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920168 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920177 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920214 5055 cpu_manager.go:414] "containerMap: removing container" podUID="97cbfd47-5703-4091-9985-dd7e40a6d611" containerName="revoke-keys" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920222 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cbfd47-5703-4091-9985-dd7e40a6d611" containerName="revoke-keys" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920245 5055 cpu_manager.go:414] "containerMap: removing container" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920252 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920265 5055 cpu_manager.go:414] "containerMap: removing container" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" Nov 02 08:39:31.920348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920272 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" Nov 02 08:39:31.920730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920454 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" Nov 02 08:39:31.920730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920474 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" Nov 02 08:39:31.920730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920484 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" containerName="main" Nov 02 08:39:31.920730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920500 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" Nov 02 08:39:31.920730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.920513 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cbfd47-5703-4091-9985-dd7e40a6d611" containerName="revoke-keys" Nov 02 08:39:31.927688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.927657 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:31.954764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.954724 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:31.954764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.954762 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr84p\" (UniqueName: \"kubernetes.io/projected/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-kube-api-access-cr84p\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:31.955010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.954777 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6af0b3b-1a28-40f8-ac81-7fe5a440f968-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:31.978368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:31.978282 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9ceb40-openshift-default-dockercfg-6d8wp\"" Nov 02 08:39:32.005136 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.005099 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl"] Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056342 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056399 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056506 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056629 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056656 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5m8b\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056741 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056775 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056858 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.062163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.056911 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181117 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181208 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5m8b\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181342 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181388 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181474 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181541 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181649 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181696 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.195226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.181804 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.205546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.205505 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.212111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.209733 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.212111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.210753 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.212111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.211461 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.213497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.213461 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.218342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.218310 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.227484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.227449 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.263990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.263895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5m8b\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.268937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.268898 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token\") pod \"e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.320600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.320537 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp"] Nov 02 08:39:32.324658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.324618 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cba6f9af/e2e-lifecycle-model-cba6f9af-kserve-587b5bc8b9-8nssp"] Nov 02 08:39:32.551278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.551187 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:32.986401 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:32.986361 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl"] Nov 02 08:39:33.019689 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:39:33.019657 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd96a9ac_a932_4d9e_914e_475702f42dd4.slice/crio-1a911acc2f8c6eb4437e3562909e783e29194bbbddd0db300595ed0a2fa41be6 WatchSource:0}: Error finding container 1a911acc2f8c6eb4437e3562909e783e29194bbbddd0db300595ed0a2fa41be6: Status 404 returned error can't find the container with id 1a911acc2f8c6eb4437e3562909e783e29194bbbddd0db300595ed0a2fa41be6 Nov 02 08:39:33.022784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.022747 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:39:33.022913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.022820 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:39:33.022913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.022848 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:39:33.096328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.096292 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6af0b3b-1a28-40f8-ac81-7fe5a440f968" path="/var/lib/kubelet/pods/c6af0b3b-1a28-40f8-ac81-7fe5a440f968/volumes" Nov 02 08:39:33.865859 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.865816 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" event={"ID":"bd96a9ac-a932-4d9e-914e-475702f42dd4","Type":"ContainerStarted","Data":"379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8"} Nov 02 08:39:33.866468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.865885 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" event={"ID":"bd96a9ac-a932-4d9e-914e-475702f42dd4","Type":"ContainerStarted","Data":"1a911acc2f8c6eb4437e3562909e783e29194bbbddd0db300595ed0a2fa41be6"} Nov 02 08:39:33.976473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:33.976359 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podStartSLOduration=2.976343724 podStartE2EDuration="2.976343724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:33.975120713 +0000 UTC m=+2021.065741135" watchObservedRunningTime="2026-11-02 08:39:33.976343724 +0000 UTC m=+2021.066964143" Nov 02 08:39:34.551708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:34.551662 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:34.557760 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:34.557731 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:34.877200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:34.877100 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:34.878233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:34.878210 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:39:39.525264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:39.525215 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz"] Nov 02 08:39:39.525817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:39.525490 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" containerID="cri-o://c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc" gracePeriod=30 Nov 02 08:39:42.101342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:42.101306 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx"] Nov 02 08:39:42.147811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:42.147770 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9eea3-wfbgx"] Nov 02 08:39:43.102751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:43.102717 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74175dd-1bbc-4800-a41f-a8a5b2d74fc4" path="/var/lib/kubelet/pods/b74175dd-1bbc-4800-a41f-a8a5b2d74fc4/volumes" Nov 02 08:39:44.551999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.551953 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 08:39:44.552441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.552068 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 08:39:44.818003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.817962 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:39:44.920680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.920639 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.920919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.920710 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.920919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.920878 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.920979 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921035 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921072 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.921290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921229 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921266 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921314 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.921398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921350 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.921398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921385 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921432 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token\") pod \"b1877c01-18c5-489c-8847-d4c15e388b02\" (UID: \"b1877c01-18c5-489c-8847-d4c15e388b02\") " Nov 02 08:39:44.921935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.921909 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.922101 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.922077 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.922101 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.922102 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.922307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.922118 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.922307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.922134 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.922450 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.922312 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:44.923688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.923655 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "kube-api-access-w4kx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:44.923796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.923695 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.924049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.924024 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:44.924175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:44.924129 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token" pod "b1877c01-18c5-489c-8847-d4c15e388b02" (UID: "b1877c01-18c5-489c-8847-d4c15e388b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:45.004581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.004540 5055 generic.go:356] "Generic (PLEG): container finished" podID="b1877c01-18c5-489c-8847-d4c15e388b02" containerID="c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc" exitCode=0 Nov 02 08:39:45.004767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.004578 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" event={"ID":"b1877c01-18c5-489c-8847-d4c15e388b02","Type":"ContainerDied","Data":"c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc"} Nov 02 08:39:45.004767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.004648 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" event={"ID":"b1877c01-18c5-489c-8847-d4c15e388b02","Type":"ContainerDied","Data":"b015db53ee881b16a5499c4e6cfa364c66f5d4908c3f353e00a71b276b11047a"} Nov 02 08:39:45.004767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.004681 5055 scope.go:122] "RemoveContainer" containerID="c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc" Nov 02 08:39:45.004974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.004890 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz" Nov 02 08:39:45.019338 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.019303 5055 scope.go:122] "RemoveContainer" containerID="c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc" Nov 02 08:39:45.019708 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:45.019671 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc\": container with ID starting with c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc not found: ID does not exist" containerID="c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc" Nov 02 08:39:45.019809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.019718 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc"} err="failed to get container status \"c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc\": rpc error: code = NotFound desc = could not find container \"c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc\": container with ID starting with c99208ff5dda31286da906a6b045d31fdcb0ec677ad9a3b5c0d1596e9483eacc not found: ID does not exist" Nov 02 08:39:45.024208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.024178 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.024337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.024216 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1877c01-18c5-489c-8847-d4c15e388b02-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.024337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.024232 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1877c01-18c5-489c-8847-d4c15e388b02-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.024337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.024246 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4kx4\" (UniqueName: \"kubernetes.io/projected/b1877c01-18c5-489c-8847-d4c15e388b02-kube-api-access-w4kx4\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.024337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.024261 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1877c01-18c5-489c-8847-d4c15e388b02-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.171650 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.171556 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz"] Nov 02 08:39:45.177017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.176981 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-4jxpz"] Nov 02 08:39:45.785049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.784989 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k"] Nov 02 08:39:45.785871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.785845 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" Nov 02 08:39:45.785871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.785872 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc7682d-cdbd-4d67-8b92-97849eb33547" containerName="istio-proxy" Nov 02 08:39:45.786056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.785905 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" Nov 02 08:39:45.786056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.785912 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" Nov 02 08:39:45.786180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.786100 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" containerName="istio-proxy" Nov 02 08:39:45.786180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.786123 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="048aa870-d17c-47ee-9ed0-583b3324f9d8" containerName="istio-proxy" Nov 02 08:39:45.791639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.791616 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:45.843476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.843442 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ceb40\"" Nov 02 08:39:45.843476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.843442 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ceb40-dockercfg-cnfld\"" Nov 02 08:39:45.851235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.851203 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k"] Nov 02 08:39:45.946228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.945273 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7"] Nov 02 08:39:46.001167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.982206 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcrc\" (UniqueName: \"kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.001167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:45.982341 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.079202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.026356 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.079202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.047157 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7"] Nov 02 08:39:46.091907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.088562 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.091907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.088732 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcrc\" (UniqueName: \"kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.091907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.091818 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.144432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.144400 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcrc\" (UniqueName: \"kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.189936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.189888 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.190295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.190259 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrkq\" (UniqueName: \"kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq\") pod \"payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.292192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.292128 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.292361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.292290 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrkq\" (UniqueName: \"kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq\") pod \"payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.293002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.292972 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.346463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.346430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrkq\" (UniqueName: \"kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq\") pod \"payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.407399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.407362 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:46.414566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.414538 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:46.784032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.783993 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7"] Nov 02 08:39:46.791538 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:39:46.791509 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43ffc68_8bcc_4851_a087_3ed4a0e0e36c.slice/crio-b3462d7962b227e076ff05523b64a65dc1295f5f254313d6c3d7def37c9e0264 WatchSource:0}: Error finding container b3462d7962b227e076ff05523b64a65dc1295f5f254313d6c3d7def37c9e0264: Status 404 returned error can't find the container with id b3462d7962b227e076ff05523b64a65dc1295f5f254313d6c3d7def37c9e0264 Nov 02 08:39:46.834941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:46.834905 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k"] Nov 02 08:39:47.093771 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.093685 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1877c01-18c5-489c-8847-d4c15e388b02" path="/var/lib/kubelet/pods/b1877c01-18c5-489c-8847-d4c15e388b02/volumes" Nov 02 08:39:47.096590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.096556 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" event={"ID":"c5891317-7be7-4998-a815-f7af30b11c2a","Type":"ContainerStarted","Data":"6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb"} Nov 02 08:39:47.096745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.096605 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" event={"ID":"c5891317-7be7-4998-a815-f7af30b11c2a","Type":"ContainerStarted","Data":"5d98f10f053f4e778aaa973b6d962686f1a7e042cf98e3b932b18b9dbea5f4f1"} Nov 02 08:39:47.096745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.096659 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:47.098313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.098286 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" event={"ID":"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c","Type":"ContainerStarted","Data":"d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379"} Nov 02 08:39:47.098401 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.098327 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" event={"ID":"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c","Type":"ContainerStarted","Data":"b3462d7962b227e076ff05523b64a65dc1295f5f254313d6c3d7def37c9e0264"} Nov 02 08:39:47.098401 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.098384 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:47.209130 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.209072 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" podStartSLOduration=2.209058407 podStartE2EDuration="2.209058407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:47.205562463 +0000 UTC m=+2034.296182881" watchObservedRunningTime="2026-11-02 08:39:47.209058407 +0000 UTC m=+2034.299678825" Nov 02 08:39:47.312989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:47.312927 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" podStartSLOduration=2.312912299 podStartE2EDuration="2.312912299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:47.31235057 +0000 UTC m=+2034.402970989" watchObservedRunningTime="2026-11-02 08:39:47.312912299 +0000 UTC m=+2034.403532717" Nov 02 08:39:49.681165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:49.681086 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb"] Nov 02 08:39:49.683193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:49.681362 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" containerID="cri-o://80bb848f054b317ec4a4e0f3107f61c0588b69b096c057a58ef89850a364f67f" gracePeriod=30 Nov 02 08:39:49.703899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:49.703860 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf"] Nov 02 08:39:49.704216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:49.704151 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" containerID="cri-o://847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5" gracePeriod=30 Nov 02 08:39:51.096525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.096484 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb"] Nov 02 08:39:51.102013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.101978 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.150219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.150181 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb"] Nov 02 08:39:51.175838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.175791 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk56t\" (UniqueName: \"kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.176025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.175848 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.176109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.176075 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.192809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.192781 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:39:51.192973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.192791 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:39:51.192973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.192919 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:39:51.197215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.197185 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:39:51.277572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.277532 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk56t\" (UniqueName: \"kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.277572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.277573 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.277837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.277654 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.278332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.278295 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.280092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.280058 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.332301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.332269 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk56t\" (UniqueName: \"kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t\") pod \"maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.419530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.419449 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:51.801067 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:39:51.801028 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd280a306_acf2_4138_96c1_2df3003424d1.slice/crio-0a5def88e1afaccf9d0883db40a552e9582cda010175aa80b0347793664d1aa1 WatchSource:0}: Error finding container 0a5def88e1afaccf9d0883db40a552e9582cda010175aa80b0347793664d1aa1: Status 404 returned error can't find the container with id 0a5def88e1afaccf9d0883db40a552e9582cda010175aa80b0347793664d1aa1 Nov 02 08:39:51.802244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:51.802101 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb"] Nov 02 08:39:52.146296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:52.146254 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" event={"ID":"d280a306-acf2-4138-96c1-2df3003424d1","Type":"ContainerStarted","Data":"a24ba88a9e4994369c56496d82b2a6525e7b5e35c5633829bd1d9ed7a0d818fa"} Nov 02 08:39:52.146698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:52.146313 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" event={"ID":"d280a306-acf2-4138-96c1-2df3003424d1","Type":"ContainerStarted","Data":"0a5def88e1afaccf9d0883db40a552e9582cda010175aa80b0347793664d1aa1"} Nov 02 08:39:52.254096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:52.254035 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" podStartSLOduration=1.254021358 podStartE2EDuration="1.254021358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:52.252783754 +0000 UTC m=+2039.343404172" watchObservedRunningTime="2026-11-02 08:39:52.254021358 +0000 UTC m=+2039.344641776" Nov 02 08:39:53.158680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:53.158636 5055 generic.go:356] "Generic (PLEG): container finished" podID="d280a306-acf2-4138-96c1-2df3003424d1" containerID="a24ba88a9e4994369c56496d82b2a6525e7b5e35c5633829bd1d9ed7a0d818fa" exitCode=0 Nov 02 08:39:53.159192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:53.158765 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" event={"ID":"d280a306-acf2-4138-96c1-2df3003424d1","Type":"ContainerDied","Data":"a24ba88a9e4994369c56496d82b2a6525e7b5e35c5633829bd1d9ed7a0d818fa"} Nov 02 08:39:54.355833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.355787 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:54.426157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.426032 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t\" (UniqueName: \"kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t\") pod \"d280a306-acf2-4138-96c1-2df3003424d1\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " Nov 02 08:39:54.426375 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.426238 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca\") pod \"d280a306-acf2-4138-96c1-2df3003424d1\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " Nov 02 08:39:54.426375 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.426321 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca\") pod \"d280a306-acf2-4138-96c1-2df3003424d1\" (UID: \"d280a306-acf2-4138-96c1-2df3003424d1\") " Nov 02 08:39:54.426782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.426754 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca" pod "d280a306-acf2-4138-96c1-2df3003424d1" (UID: "d280a306-acf2-4138-96c1-2df3003424d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:54.427078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.427054 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:54.428645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.428614 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca" pod "d280a306-acf2-4138-96c1-2df3003424d1" (UID: "d280a306-acf2-4138-96c1-2df3003424d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:54.428936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.428914 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t" pod "d280a306-acf2-4138-96c1-2df3003424d1" (UID: "d280a306-acf2-4138-96c1-2df3003424d1"). InnerVolumeSpecName "kube-api-access-nk56t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:54.529119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.529069 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a306-acf2-4138-96c1-2df3003424d1-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:54.529119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.529115 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk56t\" (UniqueName: \"kubernetes.io/projected/d280a306-acf2-4138-96c1-2df3003424d1-kube-api-access-nk56t\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:54.599865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.599819 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw"] Nov 02 08:39:54.600079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:54.600052 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" containerID="cri-o://f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b" gracePeriod=30 Nov 02 08:39:54.932266 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:54.932230 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:39:55.022599 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.022568 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:39:55.052208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.052171 5055 scope.go:122] "RemoveContainer" containerID="b19ce90bee1516120ebeb15cffcc6a2dd9a2b340ed901c8996490bac28a574b2" Nov 02 08:39:55.063678 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.063650 5055 scope.go:122] "RemoveContainer" containerID="ee7b9224e53861c4d3135128ee6729036ecb537bf0974898c06bacc082193437" Nov 02 08:39:55.075349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.075325 5055 scope.go:122] "RemoveContainer" containerID="c5287f556e2fac47805ad5ca6dd5c70552456bf9b69d7d4ce4790cccd160f9ed" Nov 02 08:39:55.150059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150015 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150079 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150118 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150204 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150268 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150312 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150366 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150395 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.150633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150446 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.150845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150808 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:55.150927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.150897 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs\") pod \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\" (UID: \"f374ffb5-7bb8-4663-9afe-f1a528ea2069\") " Nov 02 08:39:55.151187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151130 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.151481 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151456 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.151563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151518 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.151712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151682 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.151712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151707 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.151849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151723 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.151849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151741 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.151849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.151754 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.153036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.153011 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.153200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.153124 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:55.153450 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.153430 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.154502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.154476 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j" pod "f374ffb5-7bb8-4663-9afe-f1a528ea2069" (UID: "f374ffb5-7bb8-4663-9afe-f1a528ea2069"). InnerVolumeSpecName "kube-api-access-kzk4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.179943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.179908 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" Nov 02 08:39:55.179943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.179926 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb" event={"ID":"d280a306-acf2-4138-96c1-2df3003424d1","Type":"ContainerDied","Data":"0a5def88e1afaccf9d0883db40a552e9582cda010175aa80b0347793664d1aa1"} Nov 02 08:39:55.180158 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.179978 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5def88e1afaccf9d0883db40a552e9582cda010175aa80b0347793664d1aa1" Nov 02 08:39:55.181651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.181620 5055 generic.go:356] "Generic (PLEG): container finished" podID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerID="847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5" exitCode=0 Nov 02 08:39:55.181806 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.181761 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" event={"ID":"f374ffb5-7bb8-4663-9afe-f1a528ea2069","Type":"ContainerDied","Data":"847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5"} Nov 02 08:39:55.181871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.181804 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" event={"ID":"f374ffb5-7bb8-4663-9afe-f1a528ea2069","Type":"ContainerDied","Data":"60d0af447c1f97ff0116dcbc1f222f1a4493a31a7487502ab2848cc3683c1d1b"} Nov 02 08:39:55.181871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.181837 5055 scope.go:122] "RemoveContainer" containerID="847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5" Nov 02 08:39:55.181983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.181971 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf" Nov 02 08:39:55.196856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.196824 5055 generic.go:356] "Generic (PLEG): container finished" podID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerID="80bb848f054b317ec4a4e0f3107f61c0588b69b096c057a58ef89850a364f67f" exitCode=0 Nov 02 08:39:55.197019 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.196899 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" event={"ID":"83556cc2-9236-4f14-b20c-81a9ed57bdc2","Type":"ContainerDied","Data":"80bb848f054b317ec4a4e0f3107f61c0588b69b096c057a58ef89850a364f67f"} Nov 02 08:39:55.209500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.209467 5055 scope.go:122] "RemoveContainer" containerID="847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5" Nov 02 08:39:55.210488 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:39:55.210452 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5\": container with ID starting with 847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5 not found: ID does not exist" containerID="847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5" Nov 02 08:39:55.210615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.210497 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5"} err="failed to get container status \"847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5\": rpc error: code = NotFound desc = could not find container \"847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5\": container with ID starting with 847055b6fdd41699e2d4166c1f9485abf5728f42c89c34aebc77e1bf3fbab8e5 not found: ID does not exist" Nov 02 08:39:55.253866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.253824 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.253866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.253866 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.254200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.253884 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.254200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.253899 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzk4j\" (UniqueName: \"kubernetes.io/projected/f374ffb5-7bb8-4663-9afe-f1a528ea2069-kube-api-access-kzk4j\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.410829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.410798 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:39:55.458271 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.458234 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf"] Nov 02 08:39:55.467205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.467170 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-ldhbf"] Nov 02 08:39:55.499677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.499640 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn"] Nov 02 08:39:55.558584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558489 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.558584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558570 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.558820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558636 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.558820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558709 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.558820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558754 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.558820 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558790 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.559043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558821 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.559043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558863 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.559043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558951 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.559258 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.559064 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token\") pod \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\" (UID: \"83556cc2-9236-4f14-b20c-81a9ed57bdc2\") " Nov 02 08:39:55.559661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.558947 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.559661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.559210 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.559661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.559259 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:55.559661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.559379 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.560541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.560465 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.560541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.560494 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.560541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.560508 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.560861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.560815 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.560861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.560836 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.569216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.567658 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:55.569216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.567711 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "kube-api-access-7jr8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.569216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.567793 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.569216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.569024 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token" pod "83556cc2-9236-4f14-b20c-81a9ed57bdc2" (UID: "83556cc2-9236-4f14-b20c-81a9ed57bdc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.662526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.662489 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.662526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.662524 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jr8f\" (UniqueName: \"kubernetes.io/projected/83556cc2-9236-4f14-b20c-81a9ed57bdc2-kube-api-access-7jr8f\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.662747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.662539 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.662747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:55.662550 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83556cc2-9236-4f14-b20c-81a9ed57bdc2-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:56.209296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:56.209263 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" Nov 02 08:39:56.209480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:56.209299 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb" event={"ID":"83556cc2-9236-4f14-b20c-81a9ed57bdc2","Type":"ContainerDied","Data":"65a2bbe2ce61b03d4203974726893e05192530120318f01cdf9d7dba08b27405"} Nov 02 08:39:56.209480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:56.209362 5055 scope.go:122] "RemoveContainer" containerID="80bb848f054b317ec4a4e0f3107f61c0588b69b096c057a58ef89850a364f67f" Nov 02 08:39:56.365781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:56.365749 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb"] Nov 02 08:39:56.378129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:56.372478 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-8jkmb"] Nov 02 08:39:57.095962 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:57.095912 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" path="/var/lib/kubelet/pods/83556cc2-9236-4f14-b20c-81a9ed57bdc2/volumes" Nov 02 08:39:57.096699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:57.096671 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" path="/var/lib/kubelet/pods/f374ffb5-7bb8-4663-9afe-f1a528ea2069/volumes" Nov 02 08:39:58.112442 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:58.112412 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:39:58.112889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:58.112486 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:39:59.017385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:59.017329 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:39:59.374238 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:59.374086 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:39:59.374874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:59.374838 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:39:59.948345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:39:59.948304 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:40:00.138021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.137976 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138086 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138170 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138290 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138388 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138417 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138452 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:00.138525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138462 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138611 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.138767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138743 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:00.138836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.138770 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket\") pod \"d475b9a6-835f-4ac5-a681-fb2346b2976e\" (UID: \"d475b9a6-835f-4ac5-a681-fb2346b2976e\") " Nov 02 08:40:00.140246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.139265 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:00.140246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.139481 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:00.140246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.139501 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d475b9a6-835f-4ac5-a681-fb2346b2976e-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.140246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.139522 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.140246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.139536 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.140639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.140323 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:00.142728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.142697 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:00.143339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.143292 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:00.143339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.143322 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "kube-api-access-qfvwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:00.144039 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.144012 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo" pod "d475b9a6-835f-4ac5-a681-fb2346b2976e" (UID: "d475b9a6-835f-4ac5-a681-fb2346b2976e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:00.244168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.241807 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.244168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.241844 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.244168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.241859 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.244168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.241874 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvwk\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-kube-api-access-qfvwk\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.244168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.241887 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d475b9a6-835f-4ac5-a681-fb2346b2976e-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.244168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.241899 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d475b9a6-835f-4ac5-a681-fb2346b2976e-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:00.274500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.274461 5055 generic.go:356] "Generic (PLEG): container finished" podID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerID="f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b" exitCode=0 Nov 02 08:40:00.274806 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.274525 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" event={"ID":"d475b9a6-835f-4ac5-a681-fb2346b2976e","Type":"ContainerDied","Data":"f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b"} Nov 02 08:40:00.274889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.274841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" event={"ID":"d475b9a6-835f-4ac5-a681-fb2346b2976e","Type":"ContainerDied","Data":"700c50e8cb1efce39c639c437628f2e5bbd61ce82bf60bdaea5c68d16bf58868"} Nov 02 08:40:00.274889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.274866 5055 scope.go:122] "RemoveContainer" containerID="f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b" Nov 02 08:40:00.274992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.274584 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw" Nov 02 08:40:00.291002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.290976 5055 scope.go:122] "RemoveContainer" containerID="389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907" Nov 02 08:40:00.302077 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.302054 5055 scope.go:122] "RemoveContainer" containerID="f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b" Nov 02 08:40:00.302436 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:00.302416 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b\": container with ID starting with f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b not found: ID does not exist" containerID="f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b" Nov 02 08:40:00.302501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.302444 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b"} err="failed to get container status \"f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b\": rpc error: code = NotFound desc = could not find container \"f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b\": container with ID starting with f8ea09c6fa8dca2894b80c02792181c2a083238f1f1e130c4f2038a4545a917b not found: ID does not exist" Nov 02 08:40:00.302501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.302462 5055 scope.go:122] "RemoveContainer" containerID="389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907" Nov 02 08:40:00.302742 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:00.302725 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907\": container with ID starting with 389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907 not found: ID does not exist" containerID="389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907" Nov 02 08:40:00.302789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.302748 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907"} err="failed to get container status \"389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907\": rpc error: code = NotFound desc = could not find container \"389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907\": container with ID starting with 389a720967930646720a3eafef657d23465bca6fa0970b36daff0809e9bd4907 not found: ID does not exist" Nov 02 08:40:00.436818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.436670 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw"] Nov 02 08:40:00.454829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.453915 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-vp4dw"] Nov 02 08:40:00.696243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.696126 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5"] Nov 02 08:40:00.696452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.696423 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" containerID="cri-o://d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e" gracePeriod=30 Nov 02 08:40:00.712808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.712741 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq"] Nov 02 08:40:00.713588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.713518 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" containerID="cri-o://ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c" gracePeriod=30 Nov 02 08:40:00.718272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.718244 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x"] Nov 02 08:40:00.718549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.718503 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" containerID="cri-o://22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93" gracePeriod=30 Nov 02 08:40:00.723548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.723521 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7"] Nov 02 08:40:00.723778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:00.723752 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" containerID="cri-o://a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb" gracePeriod=30 Nov 02 08:40:01.092934 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:01.092886 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" path="/var/lib/kubelet/pods/d475b9a6-835f-4ac5-a681-fb2346b2976e/volumes" Nov 02 08:40:03.988133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:03.988089 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:03.988567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:03.988187 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:04.030791 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:04.030750 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:04.030964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:04.030820 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:04.652226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:04.645377 5055 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:04.652226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:04.645453 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:06.068153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.067758 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:40:06.128719 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.128689 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:40:06.135382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135350 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135396 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135439 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135501 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135576 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135696 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135719 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.135812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135772 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135910 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135884 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.135964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.135947 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket\") pod \"a11b9c5e-69f8-41a8-b476-142a0945882e\" (UID: \"a11b9c5e-69f8-41a8-b476-142a0945882e\") " Nov 02 08:40:06.136035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.136005 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.136779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.136594 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.136779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.136654 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b9c5e-69f8-41a8-b476-142a0945882e-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.136779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.136679 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.137025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.136873 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.137220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.137096 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.138932 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.138888 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "kube-api-access-7cp5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.139492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.139468 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.140307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.140282 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.140809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.140785 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo" pod "a11b9c5e-69f8-41a8-b476-142a0945882e" (UID: "a11b9c5e-69f8-41a8-b476-142a0945882e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.142604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.142582 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:40:06.152949 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.152924 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:40:06.237271 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237234 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.237476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237295 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.237476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237331 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.237476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237377 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.237476 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237407 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237663 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237723 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237726 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237762 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237797 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237798 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237853 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.237915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237900 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237928 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.237960 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238016 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238089 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238125 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238178 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238218 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238268 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238296 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238325 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238390 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.238458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238446 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238472 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy\") pod \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\" (UID: \"0dede8b3-0d74-4932-9aa0-248f41ab23ea\") " Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238483 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238506 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238560 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238647 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy\") pod \"67882b59-ab1a-4697-890d-616c30dc77df\" (UID: \"67882b59-ab1a-4697-890d-616c30dc77df\") " Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.238694 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token\") pod \"5744df3f-50af-4c01-9427-c7083bac2056\" (UID: \"5744df3f-50af-4c01-9427-c7083bac2056\") " Nov 02 08:40:06.239377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239555 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239577 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239592 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239606 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239621 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239636 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239649 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239664 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744df3f-50af-4c01-9427-c7083bac2056-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239679 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239694 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cp5m\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-kube-api-access-7cp5m\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239713 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239727 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b9c5e-69f8-41a8-b476-142a0945882e-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239615 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239654 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239781 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.239914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.239858 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.240828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.240040 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.240828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.240382 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.241358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.241332 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.242003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.241951 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.242275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.242174 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.242275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.242244 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.242609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.242568 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.244773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.244717 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.244773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.244693 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.245112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.244810 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.245112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.244925 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.245112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.244950 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "kube-api-access-9k96l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.245112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.245025 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.245876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.245840 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.246133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.246110 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token" pod "67882b59-ab1a-4697-890d-616c30dc77df" (UID: "67882b59-ab1a-4697-890d-616c30dc77df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.246133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.246125 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.246480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.246305 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms" pod "5744df3f-50af-4c01-9427-c7083bac2056" (UID: "5744df3f-50af-4c01-9427-c7083bac2056"). InnerVolumeSpecName "kube-api-access-gdwms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.246480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.246338 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j" pod "0dede8b3-0d74-4932-9aa0-248f41ab23ea" (UID: "0dede8b3-0d74-4932-9aa0-248f41ab23ea"). InnerVolumeSpecName "kube-api-access-26g7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.329356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.329258 5055 generic.go:356] "Generic (PLEG): container finished" podID="5744df3f-50af-4c01-9427-c7083bac2056" containerID="a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb" exitCode=0 Nov 02 08:40:06.329522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.329453 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" Nov 02 08:40:06.329522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.329482 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" event={"ID":"5744df3f-50af-4c01-9427-c7083bac2056","Type":"ContainerDied","Data":"a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb"} Nov 02 08:40:06.329659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.329531 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7" event={"ID":"5744df3f-50af-4c01-9427-c7083bac2056","Type":"ContainerDied","Data":"2064c036ae85ca02fca9b4cfdf5780b1eeaa30b79e0e907e9a48bc5bd212a90d"} Nov 02 08:40:06.329659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.329563 5055 scope.go:122] "RemoveContainer" containerID="a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb" Nov 02 08:40:06.336289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.336256 5055 generic.go:356] "Generic (PLEG): container finished" podID="67882b59-ab1a-4697-890d-616c30dc77df" containerID="d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e" exitCode=0 Nov 02 08:40:06.336501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.336437 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" event={"ID":"67882b59-ab1a-4697-890d-616c30dc77df","Type":"ContainerDied","Data":"d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e"} Nov 02 08:40:06.336501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.336488 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" event={"ID":"67882b59-ab1a-4697-890d-616c30dc77df","Type":"ContainerDied","Data":"6d270a3663d69678cad9c8fc4c1b741345d5a1951159d3dc8858250abc62c797"} Nov 02 08:40:06.336679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.336577 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5" Nov 02 08:40:06.343968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.343944 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.343968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.343968 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.343977 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67882b59-ab1a-4697-890d-616c30dc77df-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.343987 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.343996 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344008 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344021 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwms\" (UniqueName: \"kubernetes.io/projected/5744df3f-50af-4c01-9427-c7083bac2056-kube-api-access-gdwms\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344037 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744df3f-50af-4c01-9427-c7083bac2056-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344051 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344061 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dede8b3-0d74-4932-9aa0-248f41ab23ea-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344069 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67882b59-ab1a-4697-890d-616c30dc77df-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344081 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344089 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344097 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k96l\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-kube-api-access-9k96l\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344105 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344113 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344121 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67882b59-ab1a-4697-890d-616c30dc77df-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344130 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744df3f-50af-4c01-9427-c7083bac2056-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344150 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26g7j\" (UniqueName: \"kubernetes.io/projected/0dede8b3-0d74-4932-9aa0-248f41ab23ea-kube-api-access-26g7j\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344159 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344168 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67882b59-ab1a-4697-890d-616c30dc77df-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.344170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.344177 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dede8b3-0d74-4932-9aa0-248f41ab23ea-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.353925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.353888 5055 generic.go:356] "Generic (PLEG): container finished" podID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerID="ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c" exitCode=0 Nov 02 08:40:06.354180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.354101 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" event={"ID":"0dede8b3-0d74-4932-9aa0-248f41ab23ea","Type":"ContainerDied","Data":"ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c"} Nov 02 08:40:06.354307 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.354215 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" event={"ID":"0dede8b3-0d74-4932-9aa0-248f41ab23ea","Type":"ContainerDied","Data":"4c57ad5e42908e91f6965666001ccaa3970cb79ab64c6458544a6f0d2f441f8e"} Nov 02 08:40:06.354390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.354330 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq" Nov 02 08:40:06.358638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.358610 5055 generic.go:356] "Generic (PLEG): container finished" podID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerID="22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93" exitCode=0 Nov 02 08:40:06.358793 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.358668 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" Nov 02 08:40:06.358890 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.358688 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" event={"ID":"a11b9c5e-69f8-41a8-b476-142a0945882e","Type":"ContainerDied","Data":"22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93"} Nov 02 08:40:06.358953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.358934 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x" event={"ID":"a11b9c5e-69f8-41a8-b476-142a0945882e","Type":"ContainerDied","Data":"bdf5ca7f352b660680d3e0f6a055a61150fbfceed2fc3dba40e17da7496a1844"} Nov 02 08:40:06.365629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.364210 5055 scope.go:122] "RemoveContainer" containerID="a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb" Nov 02 08:40:06.365629 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:06.364960 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb\": container with ID starting with a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb not found: ID does not exist" containerID="a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb" Nov 02 08:40:06.365629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.365022 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb"} err="failed to get container status \"a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb\": rpc error: code = NotFound desc = could not find container \"a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb\": container with ID starting with a0df10589357ad99afcd29b990c16b9f7528b2f49cf365ee18937b6290e00dcb not found: ID does not exist" Nov 02 08:40:06.365629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.365046 5055 scope.go:122] "RemoveContainer" containerID="d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e" Nov 02 08:40:06.393799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.393766 5055 scope.go:122] "RemoveContainer" containerID="d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e" Nov 02 08:40:06.394236 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:06.394201 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e\": container with ID starting with d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e not found: ID does not exist" containerID="d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e" Nov 02 08:40:06.394354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.394246 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e"} err="failed to get container status \"d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e\": rpc error: code = NotFound desc = could not find container \"d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e\": container with ID starting with d81612258c77eb2010b4cc2b52aded06fc9f218fbfeb978d93e6ac53b56a7a4e not found: ID does not exist" Nov 02 08:40:06.394354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.394272 5055 scope.go:122] "RemoveContainer" containerID="ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c" Nov 02 08:40:06.407472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.407443 5055 scope.go:122] "RemoveContainer" containerID="eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3" Nov 02 08:40:06.420973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.420945 5055 scope.go:122] "RemoveContainer" containerID="ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c" Nov 02 08:40:06.421455 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:06.421421 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c\": container with ID starting with ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c not found: ID does not exist" containerID="ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c" Nov 02 08:40:06.421576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.421464 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c"} err="failed to get container status \"ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c\": rpc error: code = NotFound desc = could not find container \"ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c\": container with ID starting with ecca950a0cea966a4f78f51899e675dc7b0a0061ed92e70a72c6c45d858f3e5c not found: ID does not exist" Nov 02 08:40:06.421576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.421489 5055 scope.go:122] "RemoveContainer" containerID="eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3" Nov 02 08:40:06.422089 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:06.421903 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3\": container with ID starting with eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3 not found: ID does not exist" containerID="eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3" Nov 02 08:40:06.422089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.421938 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3"} err="failed to get container status \"eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3\": rpc error: code = NotFound desc = could not find container \"eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3\": container with ID starting with eb350c4239eaa246ccb4b5e8fcbc9572dcaebc6c98535ef41aba98abf7329bd3 not found: ID does not exist" Nov 02 08:40:06.422089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.421962 5055 scope.go:122] "RemoveContainer" containerID="22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93" Nov 02 08:40:06.439042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.439008 5055 scope.go:122] "RemoveContainer" containerID="22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93" Nov 02 08:40:06.439579 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:06.439507 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93\": container with ID starting with 22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93 not found: ID does not exist" containerID="22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93" Nov 02 08:40:06.439579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.439550 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93"} err="failed to get container status \"22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93\": rpc error: code = NotFound desc = could not find container \"22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93\": container with ID starting with 22bc6c4956d7c7b9dc9f2a0121b39f83370789dc5339bd44bebde01c7ee40c93 not found: ID does not exist" Nov 02 08:40:06.681382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.681294 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x"] Nov 02 08:40:06.686411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.686377 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-2fk7x"] Nov 02 08:40:06.789660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.789626 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7"] Nov 02 08:40:06.794620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.794588 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-c9zp7"] Nov 02 08:40:06.899330 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.899275 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5"] Nov 02 08:40:06.904473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:06.904433 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-5lbq5"] Nov 02 08:40:07.003663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.003621 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq"] Nov 02 08:40:07.009541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.009493 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-dwqwq"] Nov 02 08:40:07.095462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.095428 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" path="/var/lib/kubelet/pods/0dede8b3-0d74-4932-9aa0-248f41ab23ea/volumes" Nov 02 08:40:07.095956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.095942 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5744df3f-50af-4c01-9427-c7083bac2056" path="/var/lib/kubelet/pods/5744df3f-50af-4c01-9427-c7083bac2056/volumes" Nov 02 08:40:07.096390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.096376 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67882b59-ab1a-4697-890d-616c30dc77df" path="/var/lib/kubelet/pods/67882b59-ab1a-4697-890d-616c30dc77df/volumes" Nov 02 08:40:07.097118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.097104 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" path="/var/lib/kubelet/pods/a11b9c5e-69f8-41a8-b476-142a0945882e/volumes" Nov 02 08:40:07.587836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.587797 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99"] Nov 02 08:40:07.757617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:07.757582 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m"] Nov 02 08:40:10.522352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.522295 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" containerID="cri-o://88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02" gracePeriod=45 Nov 02 08:40:10.846173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.846110 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:40:10.918792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.918753 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89\" (UniqueName: \"kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89\") pod \"98c909df-aff4-4780-acf8-06d780b10794\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " Nov 02 08:40:10.919002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.918811 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home\") pod \"98c909df-aff4-4780-acf8-06d780b10794\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " Nov 02 08:40:10.919002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.918936 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir\") pod \"98c909df-aff4-4780-acf8-06d780b10794\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " Nov 02 08:40:10.919002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.918980 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache\") pod \"98c909df-aff4-4780-acf8-06d780b10794\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " Nov 02 08:40:10.919204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919019 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs\" (UniqueName: \"kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs\") pod \"98c909df-aff4-4780-acf8-06d780b10794\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " Nov 02 08:40:10.919204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919123 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm\") pod \"98c909df-aff4-4780-acf8-06d780b10794\" (UID: \"98c909df-aff4-4780-acf8-06d780b10794\") " Nov 02 08:40:10.919326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919196 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home" pod "98c909df-aff4-4780-acf8-06d780b10794" (UID: "98c909df-aff4-4780-acf8-06d780b10794"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:10.919326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919243 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir" pod "98c909df-aff4-4780-acf8-06d780b10794" (UID: "98c909df-aff4-4780-acf8-06d780b10794"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:10.919462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919430 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache" pod "98c909df-aff4-4780-acf8-06d780b10794" (UID: "98c909df-aff4-4780-acf8-06d780b10794"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:10.919800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919776 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:10.919800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919801 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:10.919995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.919816 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:10.921576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.921544 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs" pod "98c909df-aff4-4780-acf8-06d780b10794" (UID: "98c909df-aff4-4780-acf8-06d780b10794"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:10.921763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.921735 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm" pod "98c909df-aff4-4780-acf8-06d780b10794" (UID: "98c909df-aff4-4780-acf8-06d780b10794"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:10.921763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:10.921751 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89" pod "98c909df-aff4-4780-acf8-06d780b10794" (UID: "98c909df-aff4-4780-acf8-06d780b10794"). InnerVolumeSpecName "kube-api-access-nck89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:11.020989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.020946 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c909df-aff4-4780-acf8-06d780b10794-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:11.020989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.020981 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c909df-aff4-4780-acf8-06d780b10794-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:11.020989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.020993 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nck89\" (UniqueName: \"kubernetes.io/projected/98c909df-aff4-4780-acf8-06d780b10794-kube-api-access-nck89\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:11.424466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.424364 5055 generic.go:356] "Generic (PLEG): container finished" podID="98c909df-aff4-4780-acf8-06d780b10794" containerID="88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02" exitCode=0 Nov 02 08:40:11.424647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.424604 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" event={"ID":"98c909df-aff4-4780-acf8-06d780b10794","Type":"ContainerDied","Data":"88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02"} Nov 02 08:40:11.424647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.424643 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" event={"ID":"98c909df-aff4-4780-acf8-06d780b10794","Type":"ContainerDied","Data":"cf1669f45785cbe33f3555a1ebd55947b9aae2f2e36abafafb57bf1915523753"} Nov 02 08:40:11.424770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.424664 5055 scope.go:122] "RemoveContainer" containerID="88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02" Nov 02 08:40:11.424847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.424830 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn" Nov 02 08:40:11.438732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.438706 5055 scope.go:122] "RemoveContainer" containerID="88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02" Nov 02 08:40:11.439179 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:11.439120 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02\": container with ID starting with 88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02 not found: ID does not exist" containerID="88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02" Nov 02 08:40:11.439292 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.439170 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02"} err="failed to get container status \"88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02\": rpc error: code = NotFound desc = could not find container \"88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02\": container with ID starting with 88fc5727818877dcc203ed59331ee19f7e38580a53f1aaa4473ff7dcc8d7cb02 not found: ID does not exist" Nov 02 08:40:11.599215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.599169 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn"] Nov 02 08:40:11.633535 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:11.633499 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9d77b6/simulator-w5-9d77b6-kserve-5f65fcf6cc-b2qnn"] Nov 02 08:40:12.589173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:12.589088 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" podUID="4be91159-3cb5-4572-a325-33eba9d83460" containerName="payload-processing" containerID="cri-o://0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6" gracePeriod=25 Nov 02 08:40:12.758419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:12.758350 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" podUID="c04e4ecf-187b-4f5e-a92c-589ba7f17888" containerName="payload-pre-processing" containerID="cri-o://547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072" gracePeriod=25 Nov 02 08:40:12.893478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:12.893446 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:40:13.065458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.065424 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l\" (UniqueName: \"kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l\") pod \"4be91159-3cb5-4572-a325-33eba9d83460\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " Nov 02 08:40:13.065640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.065572 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume\") pod \"4be91159-3cb5-4572-a325-33eba9d83460\" (UID: \"4be91159-3cb5-4572-a325-33eba9d83460\") " Nov 02 08:40:13.066177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.066135 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume" pod "4be91159-3cb5-4572-a325-33eba9d83460" (UID: "4be91159-3cb5-4572-a325-33eba9d83460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:13.068605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.068580 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l" pod "4be91159-3cb5-4572-a325-33eba9d83460" (UID: "4be91159-3cb5-4572-a325-33eba9d83460"). InnerVolumeSpecName "kube-api-access-rwc2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:13.093606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.093576 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c909df-aff4-4780-acf8-06d780b10794" path="/var/lib/kubelet/pods/98c909df-aff4-4780-acf8-06d780b10794/volumes" Nov 02 08:40:13.099792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.099767 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:40:13.167279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.167187 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91159-3cb5-4572-a325-33eba9d83460-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:13.167279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.167215 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc2l\" (UniqueName: \"kubernetes.io/projected/4be91159-3cb5-4572-a325-33eba9d83460-kube-api-access-rwc2l\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:13.268630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.268594 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp\" (UniqueName: \"kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp\") pod \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " Nov 02 08:40:13.268630 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.268644 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume\") pod \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\" (UID: \"c04e4ecf-187b-4f5e-a92c-589ba7f17888\") " Nov 02 08:40:13.269089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.269059 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume" pod "c04e4ecf-187b-4f5e-a92c-589ba7f17888" (UID: "c04e4ecf-187b-4f5e-a92c-589ba7f17888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:13.270891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.270865 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp" pod "c04e4ecf-187b-4f5e-a92c-589ba7f17888" (UID: "c04e4ecf-187b-4f5e-a92c-589ba7f17888"). InnerVolumeSpecName "kube-api-access-vdcgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:13.370011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.369955 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdcgp\" (UniqueName: \"kubernetes.io/projected/c04e4ecf-187b-4f5e-a92c-589ba7f17888-kube-api-access-vdcgp\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:13.370011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.369992 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e4ecf-187b-4f5e-a92c-589ba7f17888-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:13.445014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.444911 5055 generic.go:356] "Generic (PLEG): container finished" podID="c04e4ecf-187b-4f5e-a92c-589ba7f17888" containerID="547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072" exitCode=0 Nov 02 08:40:13.445014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.444962 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" Nov 02 08:40:13.445014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.444996 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" event={"ID":"c04e4ecf-187b-4f5e-a92c-589ba7f17888","Type":"ContainerDied","Data":"547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072"} Nov 02 08:40:13.445299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.445056 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m" event={"ID":"c04e4ecf-187b-4f5e-a92c-589ba7f17888","Type":"ContainerDied","Data":"f00c53ee0fcd9fa9500fb01b4e9c77b510286e60c6b64355b17270b97a7b71c6"} Nov 02 08:40:13.445299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.445083 5055 scope.go:122] "RemoveContainer" containerID="547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072" Nov 02 08:40:13.446996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.446965 5055 generic.go:356] "Generic (PLEG): container finished" podID="4be91159-3cb5-4572-a325-33eba9d83460" containerID="0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6" exitCode=0 Nov 02 08:40:13.447117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.447012 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" Nov 02 08:40:13.447117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.447037 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" event={"ID":"4be91159-3cb5-4572-a325-33eba9d83460","Type":"ContainerDied","Data":"0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6"} Nov 02 08:40:13.447117 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.447084 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99" event={"ID":"4be91159-3cb5-4572-a325-33eba9d83460","Type":"ContainerDied","Data":"1fa3ec8db9536a1bb6d41701a6956c8788358825e76b10264fc53acc125b75eb"} Nov 02 08:40:13.456991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.456969 5055 scope.go:122] "RemoveContainer" containerID="547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072" Nov 02 08:40:13.457355 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:13.457333 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072\": container with ID starting with 547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072 not found: ID does not exist" containerID="547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072" Nov 02 08:40:13.457454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.457369 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072"} err="failed to get container status \"547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072\": rpc error: code = NotFound desc = could not find container \"547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072\": container with ID starting with 547e109c27e403f91f5374e6a180c991b60051724f7e2edc6c8aa24c88e4d072 not found: ID does not exist" Nov 02 08:40:13.457454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.457396 5055 scope.go:122] "RemoveContainer" containerID="0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6" Nov 02 08:40:13.467906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.467884 5055 scope.go:122] "RemoveContainer" containerID="0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6" Nov 02 08:40:13.468227 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:13.468209 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6\": container with ID starting with 0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6 not found: ID does not exist" containerID="0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6" Nov 02 08:40:13.468286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.468238 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6"} err="failed to get container status \"0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6\": rpc error: code = NotFound desc = could not find container \"0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6\": container with ID starting with 0977a921eebc174a27cbea2894822e86c64afdd1639047d2da7e5d10f6c80cf6 not found: ID does not exist" Nov 02 08:40:13.602662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.602614 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99"] Nov 02 08:40:13.607590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.607559 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-77c9b96c4c-6mb99"] Nov 02 08:40:13.705313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.705280 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m"] Nov 02 08:40:13.708555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:13.708491 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-7cb67799f-8zv8m"] Nov 02 08:40:15.093191 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:15.093157 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be91159-3cb5-4572-a325-33eba9d83460" path="/var/lib/kubelet/pods/4be91159-3cb5-4572-a325-33eba9d83460/volumes" Nov 02 08:40:15.093590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:15.093540 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04e4ecf-187b-4f5e-a92c-589ba7f17888" path="/var/lib/kubelet/pods/c04e4ecf-187b-4f5e-a92c-589ba7f17888/volumes" Nov 02 08:40:18.187032 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.186990 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845"] Nov 02 08:40:18.187779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187752 5055 cpu_manager.go:414] "containerMap: removing container" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187781 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187799 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187806 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187817 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187824 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187838 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c04e4ecf-187b-4f5e-a92c-589ba7f17888" containerName="payload-pre-processing" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187846 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04e4ecf-187b-4f5e-a92c-589ba7f17888" containerName="payload-pre-processing" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187861 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187868 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" Nov 02 08:40:18.187883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187880 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187916 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187944 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187952 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187964 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187972 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.187994 5055 cpu_manager.go:414] "containerMap: removing container" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188002 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188015 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188022 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188039 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188046 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188058 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d280a306-acf2-4138-96c1-2df3003424d1" containerName="revoke-keys" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188065 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d280a306-acf2-4138-96c1-2df3003424d1" containerName="revoke-keys" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188094 5055 cpu_manager.go:414] "containerMap: removing container" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188103 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188121 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4be91159-3cb5-4572-a325-33eba9d83460" containerName="payload-processing" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188128 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be91159-3cb5-4572-a325-33eba9d83460" containerName="payload-processing" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188300 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c909df-aff4-4780-acf8-06d780b10794" containerName="main" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188319 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be91159-3cb5-4572-a325-33eba9d83460" containerName="payload-processing" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188334 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="67882b59-ab1a-4697-890d-616c30dc77df" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188344 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188354 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188365 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04e4ecf-187b-4f5e-a92c-589ba7f17888" containerName="payload-pre-processing" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188375 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188384 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5744df3f-50af-4c01-9427-c7083bac2056" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188397 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d475b9a6-835f-4ac5-a681-fb2346b2976e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188405 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11b9c5e-69f8-41a8-b476-142a0945882e" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188416 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="83556cc2-9236-4f14-b20c-81a9ed57bdc2" containerName="istio-proxy" Nov 02 08:40:18.188411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188427 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dede8b3-0d74-4932-9aa0-248f41ab23ea" containerName="istio-proxy" Nov 02 08:40:18.189785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.188439 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d280a306-acf2-4138-96c1-2df3003424d1" containerName="revoke-keys" Nov 02 08:40:18.191926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.191905 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.233124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.233090 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.233274 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.233255 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89zv\" (UniqueName: \"kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.233326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.233314 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.241223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.241195 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:40:18.287004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.286970 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:40:18.287004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.286977 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:40:18.287004 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.286978 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:40:18.290270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.290245 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc"] Nov 02 08:40:18.291323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.291301 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f374ffb5-7bb8-4663-9afe-f1a528ea2069" containerName="istio-proxy" Nov 02 08:40:18.295248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.295224 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845"] Nov 02 08:40:18.295391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.295313 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.309163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.309111 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc"] Nov 02 08:40:18.334481 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.334444 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.334663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.334573 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.334745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.334661 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.334815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.334750 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.334880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.334845 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtr6g\" (UniqueName: \"kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.334931 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.334899 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89zv\" (UniqueName: \"kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.335496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.335468 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.336936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.336905 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.389467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.389433 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89zv\" (UniqueName: \"kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv\") pod \"maas-api-revoke-sub-cbd8a9ed-2n845\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.438208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.436681 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.438208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.436751 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtr6g\" (UniqueName: \"kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.438208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.436985 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.438208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.437904 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.442165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.441664 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.500418 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.500386 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtr6g\" (UniqueName: \"kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g\") pod \"maas-api-revoke-sub-46290c08-9wqjc\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.506425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.506393 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:18.613440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.613411 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:18.881700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.881647 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845"] Nov 02 08:40:18.884284 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:40:18.884234 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79013d27_112c_46cf_93d0_a070b2d7ff15.slice/crio-d0b1c268c44d55b7daf41e4a4f039b25510bd4351d2852ca21ba532fc72d38c9 WatchSource:0}: Error finding container d0b1c268c44d55b7daf41e4a4f039b25510bd4351d2852ca21ba532fc72d38c9: Status 404 returned error can't find the container with id d0b1c268c44d55b7daf41e4a4f039b25510bd4351d2852ca21ba532fc72d38c9 Nov 02 08:40:18.987788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:18.987752 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc"] Nov 02 08:40:19.082762 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:40:19.082722 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fe1463_eeec_4d0f_b685_7af72fac9966.slice/crio-28051821478e53195d308afa1ed79d0086aba69e18ee268c73fe2e578b9ee40b WatchSource:0}: Error finding container 28051821478e53195d308afa1ed79d0086aba69e18ee268c73fe2e578b9ee40b: Status 404 returned error can't find the container with id 28051821478e53195d308afa1ed79d0086aba69e18ee268c73fe2e578b9ee40b Nov 02 08:40:19.500965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:19.500933 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" event={"ID":"79013d27-112c-46cf-93d0-a070b2d7ff15","Type":"ContainerStarted","Data":"9c6282c71918019ecbced9bd8bd21a1abb6eb47a2af4586e5b07e335ae08a6c2"} Nov 02 08:40:19.501402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:19.500994 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" event={"ID":"79013d27-112c-46cf-93d0-a070b2d7ff15","Type":"ContainerStarted","Data":"d0b1c268c44d55b7daf41e4a4f039b25510bd4351d2852ca21ba532fc72d38c9"} Nov 02 08:40:19.502668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:19.502640 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" event={"ID":"b4fe1463-eeec-4d0f-b685-7af72fac9966","Type":"ContainerStarted","Data":"28051821478e53195d308afa1ed79d0086aba69e18ee268c73fe2e578b9ee40b"} Nov 02 08:40:19.652027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:19.651972 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" podStartSLOduration=1.651957122 podStartE2EDuration="1.651957122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:19.608596761 +0000 UTC m=+2066.699217179" watchObservedRunningTime="2026-11-02 08:40:19.651957122 +0000 UTC m=+2066.742577541" Nov 02 08:40:19.718407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:19.718348 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" podStartSLOduration=1.718333738 podStartE2EDuration="1.718333738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:19.71547261 +0000 UTC m=+2066.806093031" watchObservedRunningTime="2026-11-02 08:40:19.718333738 +0000 UTC m=+2066.808954156" Nov 02 08:40:20.008616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.008543 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql"] Nov 02 08:40:20.013591 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.013568 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.056190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.056121 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.056373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.056329 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.056373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.056368 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm9g\" (UniqueName: \"kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.070252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.070217 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql"] Nov 02 08:40:20.157220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.157176 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.157439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.157306 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.157439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.157338 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm9g\" (UniqueName: \"kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.158016 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.157994 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.159579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.159555 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.212128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.212095 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm9g\" (UniqueName: \"kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g\") pod \"maas-api-revoke-sub-17aca2a8-z92ql\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.357305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.357211 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:20.512078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.512034 5055 generic.go:356] "Generic (PLEG): container finished" podID="79013d27-112c-46cf-93d0-a070b2d7ff15" containerID="9c6282c71918019ecbced9bd8bd21a1abb6eb47a2af4586e5b07e335ae08a6c2" exitCode=0 Nov 02 08:40:20.512555 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.512124 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" event={"ID":"79013d27-112c-46cf-93d0-a070b2d7ff15","Type":"ContainerDied","Data":"9c6282c71918019ecbced9bd8bd21a1abb6eb47a2af4586e5b07e335ae08a6c2"} Nov 02 08:40:20.513971 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.513943 5055 generic.go:356] "Generic (PLEG): container finished" podID="b4fe1463-eeec-4d0f-b685-7af72fac9966" containerID="216a8595bc5c8dbad53aa5b78a657258140b70a8cedd03234d63e18215ab987d" exitCode=0 Nov 02 08:40:20.514121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.513980 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" event={"ID":"b4fe1463-eeec-4d0f-b685-7af72fac9966","Type":"ContainerDied","Data":"216a8595bc5c8dbad53aa5b78a657258140b70a8cedd03234d63e18215ab987d"} Nov 02 08:40:20.732051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:20.732000 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql"] Nov 02 08:40:20.733254 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:40:20.733212 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b5a7c5_1bf4_4fac_bb12_960a88cbd748.slice/crio-eb63e0d2ac5fef1f3f5fa2a983b95b145868603f3f3095a2f10133aa50634ebd WatchSource:0}: Error finding container eb63e0d2ac5fef1f3f5fa2a983b95b145868603f3f3095a2f10133aa50634ebd: Status 404 returned error can't find the container with id eb63e0d2ac5fef1f3f5fa2a983b95b145868603f3f3095a2f10133aa50634ebd Nov 02 08:40:21.524812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.524770 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" event={"ID":"17b5a7c5-1bf4-4fac-bb12-960a88cbd748","Type":"ContainerStarted","Data":"0738ebacf46677795c3aff032f8ce2e3cd61fc548703bbe9c7546b47c66e4bae"} Nov 02 08:40:21.525342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.524831 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" event={"ID":"17b5a7c5-1bf4-4fac-bb12-960a88cbd748","Type":"ContainerStarted","Data":"eb63e0d2ac5fef1f3f5fa2a983b95b145868603f3f3095a2f10133aa50634ebd"} Nov 02 08:40:21.637228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.636740 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" podStartSLOduration=2.636717442 podStartE2EDuration="2.636717442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:21.633048409 +0000 UTC m=+2068.723668828" watchObservedRunningTime="2026-11-02 08:40:21.636717442 +0000 UTC m=+2068.727337862" Nov 02 08:40:21.730672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.730641 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:21.776413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.776344 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca\") pod \"b4fe1463-eeec-4d0f-b685-7af72fac9966\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " Nov 02 08:40:21.776522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.776480 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g\" (UniqueName: \"kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g\") pod \"b4fe1463-eeec-4d0f-b685-7af72fac9966\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " Nov 02 08:40:21.776567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.776550 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca\") pod \"b4fe1463-eeec-4d0f-b685-7af72fac9966\" (UID: \"b4fe1463-eeec-4d0f-b685-7af72fac9966\") " Nov 02 08:40:21.776733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.776705 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca" pod "b4fe1463-eeec-4d0f-b685-7af72fac9966" (UID: "b4fe1463-eeec-4d0f-b685-7af72fac9966"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:21.776931 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.776913 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:21.778375 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.778350 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g" pod "b4fe1463-eeec-4d0f-b685-7af72fac9966" (UID: "b4fe1463-eeec-4d0f-b685-7af72fac9966"). InnerVolumeSpecName "kube-api-access-gtr6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:21.778462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.778422 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca" pod "b4fe1463-eeec-4d0f-b685-7af72fac9966" (UID: "b4fe1463-eeec-4d0f-b685-7af72fac9966"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:21.827268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.827241 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:21.877766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.877730 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca\") pod \"79013d27-112c-46cf-93d0-a070b2d7ff15\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " Nov 02 08:40:21.877981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.877832 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv\" (UniqueName: \"kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv\") pod \"79013d27-112c-46cf-93d0-a070b2d7ff15\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " Nov 02 08:40:21.877981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.877894 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca\") pod \"79013d27-112c-46cf-93d0-a070b2d7ff15\" (UID: \"79013d27-112c-46cf-93d0-a070b2d7ff15\") " Nov 02 08:40:21.878331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.878296 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca" pod "79013d27-112c-46cf-93d0-a070b2d7ff15" (UID: "79013d27-112c-46cf-93d0-a070b2d7ff15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:21.878552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.878523 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtr6g\" (UniqueName: \"kubernetes.io/projected/b4fe1463-eeec-4d0f-b685-7af72fac9966-kube-api-access-gtr6g\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:21.878552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.878545 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe1463-eeec-4d0f-b685-7af72fac9966-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:21.878697 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.878561 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:21.880057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.880036 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca" pod "79013d27-112c-46cf-93d0-a070b2d7ff15" (UID: "79013d27-112c-46cf-93d0-a070b2d7ff15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:21.880587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.880561 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv" pod "79013d27-112c-46cf-93d0-a070b2d7ff15" (UID: "79013d27-112c-46cf-93d0-a070b2d7ff15"). InnerVolumeSpecName "kube-api-access-m89zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:21.979746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.979711 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89zv\" (UniqueName: \"kubernetes.io/projected/79013d27-112c-46cf-93d0-a070b2d7ff15-kube-api-access-m89zv\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:21.979746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:21.979743 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79013d27-112c-46cf-93d0-a070b2d7ff15-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:22.538518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.538488 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" Nov 02 08:40:22.539028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.538489 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845" event={"ID":"79013d27-112c-46cf-93d0-a070b2d7ff15","Type":"ContainerDied","Data":"d0b1c268c44d55b7daf41e4a4f039b25510bd4351d2852ca21ba532fc72d38c9"} Nov 02 08:40:22.539028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.538616 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0b1c268c44d55b7daf41e4a4f039b25510bd4351d2852ca21ba532fc72d38c9" Nov 02 08:40:22.540071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.540048 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" Nov 02 08:40:22.540245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.540093 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc" event={"ID":"b4fe1463-eeec-4d0f-b685-7af72fac9966","Type":"ContainerDied","Data":"28051821478e53195d308afa1ed79d0086aba69e18ee268c73fe2e578b9ee40b"} Nov 02 08:40:22.540245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.540173 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28051821478e53195d308afa1ed79d0086aba69e18ee268c73fe2e578b9ee40b" Nov 02 08:40:22.542017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.541986 5055 generic.go:356] "Generic (PLEG): container finished" podID="17b5a7c5-1bf4-4fac-bb12-960a88cbd748" containerID="0738ebacf46677795c3aff032f8ce2e3cd61fc548703bbe9c7546b47c66e4bae" exitCode=0 Nov 02 08:40:22.542132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:22.542074 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" event={"ID":"17b5a7c5-1bf4-4fac-bb12-960a88cbd748","Type":"ContainerDied","Data":"0738ebacf46677795c3aff032f8ce2e3cd61fc548703bbe9c7546b47c66e4bae"} Nov 02 08:40:23.721852 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.721818 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:23.804192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.804087 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca\") pod \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " Nov 02 08:40:23.804364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.804347 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca\") pod \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " Nov 02 08:40:23.804451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.804432 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g\" (UniqueName: \"kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g\") pod \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\" (UID: \"17b5a7c5-1bf4-4fac-bb12-960a88cbd748\") " Nov 02 08:40:23.804582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.804543 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca" pod "17b5a7c5-1bf4-4fac-bb12-960a88cbd748" (UID: "17b5a7c5-1bf4-4fac-bb12-960a88cbd748"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:23.805005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.804984 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:23.806684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.806646 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca" pod "17b5a7c5-1bf4-4fac-bb12-960a88cbd748" (UID: "17b5a7c5-1bf4-4fac-bb12-960a88cbd748"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:23.806954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.806914 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g" pod "17b5a7c5-1bf4-4fac-bb12-960a88cbd748" (UID: "17b5a7c5-1bf4-4fac-bb12-960a88cbd748"). InnerVolumeSpecName "kube-api-access-tgm9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:23.909823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.907596 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgm9g\" (UniqueName: \"kubernetes.io/projected/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-kube-api-access-tgm9g\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:23.909823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:23.907639 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b5a7c5-1bf4-4fac-bb12-960a88cbd748-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:24.558847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:24.558808 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" event={"ID":"17b5a7c5-1bf4-4fac-bb12-960a88cbd748","Type":"ContainerDied","Data":"eb63e0d2ac5fef1f3f5fa2a983b95b145868603f3f3095a2f10133aa50634ebd"} Nov 02 08:40:24.558847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:24.558848 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql" Nov 02 08:40:24.559074 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:24.558872 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb63e0d2ac5fef1f3f5fa2a983b95b145868603f3f3095a2f10133aa50634ebd" Nov 02 08:40:24.923634 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:24.923516 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:40:28.051986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.051942 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6"] Nov 02 08:40:28.052545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052523 5055 cpu_manager.go:414] "containerMap: removing container" podUID="17b5a7c5-1bf4-4fac-bb12-960a88cbd748" containerName="revoke-keys" Nov 02 08:40:28.052545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052541 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b5a7c5-1bf4-4fac-bb12-960a88cbd748" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052561 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b4fe1463-eeec-4d0f-b685-7af72fac9966" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052568 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fe1463-eeec-4d0f-b685-7af72fac9966" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052594 5055 cpu_manager.go:414] "containerMap: removing container" podUID="79013d27-112c-46cf-93d0-a070b2d7ff15" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052600 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79013d27-112c-46cf-93d0-a070b2d7ff15" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052694 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="79013d27-112c-46cf-93d0-a070b2d7ff15" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052712 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b5a7c5-1bf4-4fac-bb12-960a88cbd748" containerName="revoke-keys" Nov 02 08:40:28.052770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.052720 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fe1463-eeec-4d0f-b685-7af72fac9966" containerName="revoke-keys" Nov 02 08:40:28.056449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.056426 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.105412 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.105379 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:40:28.151301 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.151261 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:40:28.151508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.151398 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:40:28.151670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.151648 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:40:28.157046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.157014 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4"] Nov 02 08:40:28.161362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.161333 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.161523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.161504 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjm8p\" (UniqueName: \"kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.161708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.161686 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.162114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.162091 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.162298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.162274 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6"] Nov 02 08:40:28.208105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.208066 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4"] Nov 02 08:40:28.262407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.262369 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjm8p\" (UniqueName: \"kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.262608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.262439 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.262608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.262569 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.262729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.262613 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577dg\" (UniqueName: \"kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.262729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.262640 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.262857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.262835 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.263343 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.263326 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.265326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.265305 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.316605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.316509 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjm8p\" (UniqueName: \"kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p\") pod \"maas-api-revoke-sub-5d55b990-dw7c6\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.363704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.363653 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.363891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.363757 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.363891 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.363786 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577dg\" (UniqueName: \"kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.364478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.364447 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.366609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.366585 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.369929 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.369907 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:28.418455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.418419 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577dg\" (UniqueName: \"kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg\") pod \"maas-api-revoke-sub-df5c6fe0-47qm4\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.479898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.479866 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:28.760355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.760316 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6"] Nov 02 08:40:28.888089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:28.888052 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4"] Nov 02 08:40:28.888436 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:40:28.888404 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727c2ca1_0467_4b12_900b_86cab652c148.slice/crio-0e5993d4293e26ee6bc64095cb97ba318a09e177080387a20558ed3fcb34e0d7 WatchSource:0}: Error finding container 0e5993d4293e26ee6bc64095cb97ba318a09e177080387a20558ed3fcb34e0d7: Status 404 returned error can't find the container with id 0e5993d4293e26ee6bc64095cb97ba318a09e177080387a20558ed3fcb34e0d7 Nov 02 08:40:29.607543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.607493 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" event={"ID":"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2","Type":"ContainerStarted","Data":"ad89f084385c95e41116b0f674feaad95f236f1d073313120c285b84699fe505"} Nov 02 08:40:29.608361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.607569 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" event={"ID":"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2","Type":"ContainerStarted","Data":"2b6fec4dbff692b353b14dfc3f139246349b53eb06bd87f9a0f041ddcb156363"} Nov 02 08:40:29.609735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.609695 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" event={"ID":"727c2ca1-0467-4b12-900b-86cab652c148","Type":"ContainerStarted","Data":"498b15ba644acc9c560e60be1030635d41ab98c01749dada37647b372661456b"} Nov 02 08:40:29.609866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.609756 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" event={"ID":"727c2ca1-0467-4b12-900b-86cab652c148","Type":"ContainerStarted","Data":"0e5993d4293e26ee6bc64095cb97ba318a09e177080387a20558ed3fcb34e0d7"} Nov 02 08:40:29.715413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.715354 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" podStartSLOduration=1.7153395489999999 podStartE2EDuration="1.715339549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:29.713247922 +0000 UTC m=+2076.803868341" watchObservedRunningTime="2026-11-02 08:40:29.715339549 +0000 UTC m=+2076.805959965" Nov 02 08:40:29.802715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.802675 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j"] Nov 02 08:40:29.807493 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.807457 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.821072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.821014 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" podStartSLOduration=1.8210002950000002 podStartE2EDuration="1.821000295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:29.819362326 +0000 UTC m=+2076.909982745" watchObservedRunningTime="2026-11-02 08:40:29.821000295 +0000 UTC m=+2076.911620714" Nov 02 08:40:29.865912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.865810 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j"] Nov 02 08:40:29.888007 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.887965 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.888346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.888315 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.888474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.888446 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pht22\" (UniqueName: \"kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.989711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.989668 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pht22\" (UniqueName: \"kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.989933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.989841 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.989933 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.989867 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.990554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.990526 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:29.992364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:29.992342 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:30.043685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.043647 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pht22\" (UniqueName: \"kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:30.127807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.127707 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:30.504377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.504341 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j"] Nov 02 08:40:30.506295 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:40:30.506257 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0428c4df_62ff_471f_9a9f_f4c778c43166.slice/crio-c33a06968cd88436f80f7fd1dd22f580528a82457506349bc31882e81758ce8c WatchSource:0}: Error finding container c33a06968cd88436f80f7fd1dd22f580528a82457506349bc31882e81758ce8c: Status 404 returned error can't find the container with id c33a06968cd88436f80f7fd1dd22f580528a82457506349bc31882e81758ce8c Nov 02 08:40:30.620856 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.620817 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" event={"ID":"0428c4df-62ff-471f-9a9f-f4c778c43166","Type":"ContainerStarted","Data":"c33a06968cd88436f80f7fd1dd22f580528a82457506349bc31882e81758ce8c"} Nov 02 08:40:30.622828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.622800 5055 generic.go:356] "Generic (PLEG): container finished" podID="a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" containerID="ad89f084385c95e41116b0f674feaad95f236f1d073313120c285b84699fe505" exitCode=0 Nov 02 08:40:30.622973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.622894 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" event={"ID":"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2","Type":"ContainerDied","Data":"ad89f084385c95e41116b0f674feaad95f236f1d073313120c285b84699fe505"} Nov 02 08:40:30.624943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.624919 5055 generic.go:356] "Generic (PLEG): container finished" podID="727c2ca1-0467-4b12-900b-86cab652c148" containerID="498b15ba644acc9c560e60be1030635d41ab98c01749dada37647b372661456b" exitCode=0 Nov 02 08:40:30.625054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:30.624957 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" event={"ID":"727c2ca1-0467-4b12-900b-86cab652c148","Type":"ContainerDied","Data":"498b15ba644acc9c560e60be1030635d41ab98c01749dada37647b372661456b"} Nov 02 08:40:31.635386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.635299 5055 generic.go:356] "Generic (PLEG): container finished" podID="0428c4df-62ff-471f-9a9f-f4c778c43166" containerID="2cdff706f6df9f38d09f8a26c7877749298c48c78f1b1bee0f5f62b4d2d1316b" exitCode=0 Nov 02 08:40:31.635782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.635379 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" event={"ID":"0428c4df-62ff-471f-9a9f-f4c778c43166","Type":"ContainerDied","Data":"2cdff706f6df9f38d09f8a26c7877749298c48c78f1b1bee0f5f62b4d2d1316b"} Nov 02 08:40:31.857504 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.857465 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:31.908417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.908341 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:31.922046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.922004 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p\" (UniqueName: \"kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p\") pod \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " Nov 02 08:40:31.923223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.923196 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca\") pod \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " Nov 02 08:40:31.923397 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.923339 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca\") pod \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\" (UID: \"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2\") " Nov 02 08:40:31.923605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.923578 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca" pod "a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" (UID: "a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:31.923824 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.923805 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:31.924728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.924698 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p" pod "a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" (UID: "a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2"). InnerVolumeSpecName "kube-api-access-gjm8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:31.925502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:31.925478 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca" pod "a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" (UID: "a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:32.042165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.040933 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca\") pod \"727c2ca1-0467-4b12-900b-86cab652c148\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " Nov 02 08:40:32.042165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.041009 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg\" (UniqueName: \"kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg\") pod \"727c2ca1-0467-4b12-900b-86cab652c148\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " Nov 02 08:40:32.042165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.041132 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca\") pod \"727c2ca1-0467-4b12-900b-86cab652c148\" (UID: \"727c2ca1-0467-4b12-900b-86cab652c148\") " Nov 02 08:40:32.042500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.042401 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:32.042500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.042428 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjm8p\" (UniqueName: \"kubernetes.io/projected/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2-kube-api-access-gjm8p\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:32.042980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.042819 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca" pod "727c2ca1-0467-4b12-900b-86cab652c148" (UID: "727c2ca1-0467-4b12-900b-86cab652c148"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:32.046120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.046094 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca" pod "727c2ca1-0467-4b12-900b-86cab652c148" (UID: "727c2ca1-0467-4b12-900b-86cab652c148"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:32.054473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.054414 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg" pod "727c2ca1-0467-4b12-900b-86cab652c148" (UID: "727c2ca1-0467-4b12-900b-86cab652c148"). InnerVolumeSpecName "kube-api-access-577dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:32.143783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.143743 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:32.143783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.143776 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577dg\" (UniqueName: \"kubernetes.io/projected/727c2ca1-0467-4b12-900b-86cab652c148-kube-api-access-577dg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:32.143783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.143789 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727c2ca1-0467-4b12-900b-86cab652c148-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:32.646950 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.646914 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" Nov 02 08:40:32.647499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.646914 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6" event={"ID":"a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2","Type":"ContainerDied","Data":"2b6fec4dbff692b353b14dfc3f139246349b53eb06bd87f9a0f041ddcb156363"} Nov 02 08:40:32.647499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.647109 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6fec4dbff692b353b14dfc3f139246349b53eb06bd87f9a0f041ddcb156363" Nov 02 08:40:32.648751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.648727 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" Nov 02 08:40:32.648888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.648761 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4" event={"ID":"727c2ca1-0467-4b12-900b-86cab652c148","Type":"ContainerDied","Data":"0e5993d4293e26ee6bc64095cb97ba318a09e177080387a20558ed3fcb34e0d7"} Nov 02 08:40:32.648888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.648809 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e5993d4293e26ee6bc64095cb97ba318a09e177080387a20558ed3fcb34e0d7" Nov 02 08:40:32.822224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.822194 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:32.954570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.954535 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca\") pod \"0428c4df-62ff-471f-9a9f-f4c778c43166\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " Nov 02 08:40:32.954783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.954675 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca\") pod \"0428c4df-62ff-471f-9a9f-f4c778c43166\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " Nov 02 08:40:32.954783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.954714 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22\" (UniqueName: \"kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22\") pod \"0428c4df-62ff-471f-9a9f-f4c778c43166\" (UID: \"0428c4df-62ff-471f-9a9f-f4c778c43166\") " Nov 02 08:40:32.955017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.954987 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca" pod "0428c4df-62ff-471f-9a9f-f4c778c43166" (UID: "0428c4df-62ff-471f-9a9f-f4c778c43166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:32.955404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.955375 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:32.957803 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.957773 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca" pod "0428c4df-62ff-471f-9a9f-f4c778c43166" (UID: "0428c4df-62ff-471f-9a9f-f4c778c43166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:32.958371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:32.958339 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22" pod "0428c4df-62ff-471f-9a9f-f4c778c43166" (UID: "0428c4df-62ff-471f-9a9f-f4c778c43166"). InnerVolumeSpecName "kube-api-access-pht22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:33.056707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:33.056664 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0428c4df-62ff-471f-9a9f-f4c778c43166-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:33.056707 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:33.056701 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pht22\" (UniqueName: \"kubernetes.io/projected/0428c4df-62ff-471f-9a9f-f4c778c43166-kube-api-access-pht22\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:33.659413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:33.659383 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" Nov 02 08:40:33.659846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:33.659382 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j" event={"ID":"0428c4df-62ff-471f-9a9f-f4c778c43166","Type":"ContainerDied","Data":"c33a06968cd88436f80f7fd1dd22f580528a82457506349bc31882e81758ce8c"} Nov 02 08:40:33.659846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:33.659513 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33a06968cd88436f80f7fd1dd22f580528a82457506349bc31882e81758ce8c" Nov 02 08:40:43.131256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131218 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd"] Nov 02 08:40:43.131783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131770 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" containerName="revoke-keys" Nov 02 08:40:43.131825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131785 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" containerName="revoke-keys" Nov 02 08:40:43.131825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131799 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0428c4df-62ff-471f-9a9f-f4c778c43166" containerName="revoke-keys" Nov 02 08:40:43.131825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131804 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0428c4df-62ff-471f-9a9f-f4c778c43166" containerName="revoke-keys" Nov 02 08:40:43.131825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131818 5055 cpu_manager.go:414] "containerMap: removing container" podUID="727c2ca1-0467-4b12-900b-86cab652c148" containerName="revoke-keys" Nov 02 08:40:43.131825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.131823 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727c2ca1-0467-4b12-900b-86cab652c148" containerName="revoke-keys" Nov 02 08:40:43.132043 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.132027 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" containerName="revoke-keys" Nov 02 08:40:43.132099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.132048 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0428c4df-62ff-471f-9a9f-f4c778c43166" containerName="revoke-keys" Nov 02 08:40:43.132099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.132062 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="727c2ca1-0467-4b12-900b-86cab652c148" containerName="revoke-keys" Nov 02 08:40:43.135903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.135883 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.198546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.198511 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ad4580-openshift-default-dockercfg-tj5ft\"" Nov 02 08:40:43.213687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.213623 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.213919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.213899 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.214061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.214046 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.214298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.214280 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.214471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.214455 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82lfx\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.214596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.214583 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.214708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.214693 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.214872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.214858 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.215055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.215042 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.255443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.255401 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd"] Nov 02 08:40:43.316894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.316850 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.316908 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.316942 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317036 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317098 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317089 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82lfx\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317131 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317203 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317256 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.317381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317340 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.322170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.317971 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.322170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.318303 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.322170 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.319522 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.328532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.328498 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.331054 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.331023 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.346694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.346634 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.348942 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.348898 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.372730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.372698 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82lfx\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.378225 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.378190 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token\") pod \"e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.449217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.449185 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:43.872108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.872069 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd"] Nov 02 08:40:43.873180 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:40:43.873124 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695546c1_9771_4a9b_a2bd_23ef70aa73f5.slice/crio-efe31fa6e96f1db3c328a0fc9cfaa759098c95b7256092f326029f5e551ca081 WatchSource:0}: Error finding container efe31fa6e96f1db3c328a0fc9cfaa759098c95b7256092f326029f5e551ca081: Status 404 returned error can't find the container with id efe31fa6e96f1db3c328a0fc9cfaa759098c95b7256092f326029f5e551ca081 Nov 02 08:40:43.875828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.875791 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:40:43.875946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.875878 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:40:43.875946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:43.875928 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:40:44.814802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:44.814755 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" event={"ID":"695546c1-9771-4a9b-a2bd-23ef70aa73f5","Type":"ContainerStarted","Data":"d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c"} Nov 02 08:40:44.815318 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:44.814825 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" event={"ID":"695546c1-9771-4a9b-a2bd-23ef70aa73f5","Type":"ContainerStarted","Data":"efe31fa6e96f1db3c328a0fc9cfaa759098c95b7256092f326029f5e551ca081"} Nov 02 08:40:44.966810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:44.962435 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podStartSLOduration=1.962409637 podStartE2EDuration="1.962409637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:44.948687448 +0000 UTC m=+2092.039307868" watchObservedRunningTime="2026-11-02 08:40:44.962409637 +0000 UTC m=+2092.053030059" Nov 02 08:40:45.449562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:45.449530 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:45.456668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:45.456641 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:45.822836 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:45.822761 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:45.823804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:45.823781 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:40:51.965029 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:51.964992 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j"] Nov 02 08:40:52.010289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:52.010248 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a9bb-7np2j"] Nov 02 08:40:53.094658 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:53.094626 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0428c4df-62ff-471f-9a9f-f4c778c43166" path="/var/lib/kubelet/pods/0428c4df-62ff-471f-9a9f-f4c778c43166/volumes" Nov 02 08:40:53.778239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:53.778184 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:40:53.779152 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:53.779116 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:40:54.922241 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:40:54.922199 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:40:55.192928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:55.179257 5055 scope.go:122] "RemoveContainer" containerID="be0a089d14f0acda65bae694d1b6abd2fcc764b10391403552295fa60049fcdf" Nov 02 08:40:58.909172 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:58.909123 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf"] Nov 02 08:40:58.909563 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:58.909393 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" containerID="cri-o://7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489" gracePeriod=30 Nov 02 08:40:59.678112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.678067 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s"] Nov 02 08:40:59.684521 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.684491 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.723489 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.723450 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.723675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.723505 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhckw\" (UniqueName: \"kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.723675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.723603 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.733223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.733185 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:40:59.733387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.733368 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:40:59.736104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.736086 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:40:59.740084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.740058 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s"] Nov 02 08:40:59.825324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.825283 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.825539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.825339 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhckw\" (UniqueName: \"kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.825539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.825378 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.826024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.826000 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.827968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.827929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.833663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.833629 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:40:59.878382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.878344 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhckw\" (UniqueName: \"kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw\") pod \"maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:40:59.999261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:40:59.999222 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:41:00.376350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:00.376320 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s"] Nov 02 08:41:00.379706 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:41:00.379676 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6cdac9_cbb9_4fa6_8c06_817564ee9be5.slice/crio-eb066a80a0c1e7d02d955e79ac0569a96d041da5b66f0a3c17a396f7a700e47c WatchSource:0}: Error finding container eb066a80a0c1e7d02d955e79ac0569a96d041da5b66f0a3c17a396f7a700e47c: Status 404 returned error can't find the container with id eb066a80a0c1e7d02d955e79ac0569a96d041da5b66f0a3c17a396f7a700e47c Nov 02 08:41:00.938835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:00.938798 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" event={"ID":"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5","Type":"ContainerStarted","Data":"b0eb58cf2eea58d0368d12c7110ebbc55c357c11fb78e911c409f29d019469be"} Nov 02 08:41:00.939051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:00.938856 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" event={"ID":"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5","Type":"ContainerStarted","Data":"eb066a80a0c1e7d02d955e79ac0569a96d041da5b66f0a3c17a396f7a700e47c"} Nov 02 08:41:01.048730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:01.048677 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" podStartSLOduration=2.048664264 podStartE2EDuration="2.048664264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:01.046902961 +0000 UTC m=+2108.137523379" watchObservedRunningTime="2026-11-02 08:41:01.048664264 +0000 UTC m=+2108.139284682" Nov 02 08:41:01.243543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:01.243495 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:41:01.243747 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:01.243558 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:41:01.947874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:01.947842 5055 generic.go:356] "Generic (PLEG): container finished" podID="2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" containerID="b0eb58cf2eea58d0368d12c7110ebbc55c357c11fb78e911c409f29d019469be" exitCode=0 Nov 02 08:41:01.948066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:01.947888 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" event={"ID":"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5","Type":"ContainerDied","Data":"b0eb58cf2eea58d0368d12c7110ebbc55c357c11fb78e911c409f29d019469be"} Nov 02 08:41:03.116211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.116181 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:41:03.169468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.169428 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw\" (UniqueName: \"kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw\") pod \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " Nov 02 08:41:03.169644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.169532 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca\") pod \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " Nov 02 08:41:03.169644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.169579 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca\") pod \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\" (UID: \"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5\") " Nov 02 08:41:03.169945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.169915 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca" pod "2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" (UID: "2c6cdac9-cbb9-4fa6-8c06-817564ee9be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:03.170286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.170264 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:03.171802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.171774 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca" pod "2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" (UID: "2c6cdac9-cbb9-4fa6-8c06-817564ee9be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:41:03.171902 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.171860 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw" pod "2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" (UID: "2c6cdac9-cbb9-4fa6-8c06-817564ee9be5"). InnerVolumeSpecName "kube-api-access-zhckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:03.271203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.271092 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:03.271203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.271126 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhckw\" (UniqueName: \"kubernetes.io/projected/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5-kube-api-access-zhckw\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:03.963923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.963897 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" Nov 02 08:41:03.964067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.963931 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s" event={"ID":"2c6cdac9-cbb9-4fa6-8c06-817564ee9be5","Type":"ContainerDied","Data":"eb066a80a0c1e7d02d955e79ac0569a96d041da5b66f0a3c17a396f7a700e47c"} Nov 02 08:41:03.964067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:03.963982 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb066a80a0c1e7d02d955e79ac0569a96d041da5b66f0a3c17a396f7a700e47c" Nov 02 08:41:04.194299 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.194236 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:41:04.283249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283179 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283236 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283286 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283324 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283453 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283481 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.283611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283550 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283597 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283639 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283660 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.283906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283711 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy\") pod \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\" (UID: \"fcf55650-e5c8-49d1-b9ab-fcd5887c905b\") " Nov 02 08:41:04.283906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.283809 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.284352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.284287 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.284352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.284328 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.284352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.284344 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.284352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.284302 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:04.286662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.284773 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.286662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.286455 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:04.286810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.286661 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:41:04.287089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.287057 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "kube-api-access-fm6cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:04.292473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.292446 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy" pod "fcf55650-e5c8-49d1-b9ab-fcd5887c905b" (UID: "fcf55650-e5c8-49d1-b9ab-fcd5887c905b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.385885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.385849 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm6cq\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-kube-api-access-fm6cq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.385885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.385881 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.385885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.385892 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.386287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.385901 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.386287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.385910 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.386287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.385917 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf55650-e5c8-49d1-b9ab-fcd5887c905b-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.973815 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.973773 5055 generic.go:356] "Generic (PLEG): container finished" podID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerID="7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489" exitCode=0 Nov 02 08:41:04.974018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.973832 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" Nov 02 08:41:04.974018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.973856 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" event={"ID":"fcf55650-e5c8-49d1-b9ab-fcd5887c905b","Type":"ContainerDied","Data":"7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489"} Nov 02 08:41:04.974018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.973913 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf" event={"ID":"fcf55650-e5c8-49d1-b9ab-fcd5887c905b","Type":"ContainerDied","Data":"abd4cf1d486d72a4c9fa172900437a9173042ae3a2e9d381701dc5b85ac7dfd3"} Nov 02 08:41:04.974018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.973971 5055 scope.go:122] "RemoveContainer" containerID="7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489" Nov 02 08:41:04.986486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.986460 5055 scope.go:122] "RemoveContainer" containerID="7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489" Nov 02 08:41:04.986903 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:41:04.986880 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489\": container with ID starting with 7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489 not found: ID does not exist" containerID="7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489" Nov 02 08:41:04.986987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:04.986912 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489"} err="failed to get container status \"7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489\": rpc error: code = NotFound desc = could not find container \"7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489\": container with ID starting with 7d28f76b68ea62f3be0604055cbd5576165ba7470146a42cc8f3f475ae99a489 not found: ID does not exist" Nov 02 08:41:05.128070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:05.128029 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf"] Nov 02 08:41:05.174472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:05.174437 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-kkvwf"] Nov 02 08:41:07.093666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:07.093629 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" path="/var/lib/kubelet/pods/fcf55650-e5c8-49d1-b9ab-fcd5887c905b/volumes" Nov 02 08:41:16.440243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.440202 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk"] Nov 02 08:41:16.441192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.441162 5055 cpu_manager.go:414] "containerMap: removing container" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" Nov 02 08:41:16.441192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.441191 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" Nov 02 08:41:16.441456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.441240 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" containerName="revoke-keys" Nov 02 08:41:16.441456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.441249 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" containerName="revoke-keys" Nov 02 08:41:16.441456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.441450 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf55650-e5c8-49d1-b9ab-fcd5887c905b" containerName="istio-proxy" Nov 02 08:41:16.441628 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.441472 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" containerName="revoke-keys" Nov 02 08:41:16.446712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.446690 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.487257 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.486947 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv"] Nov 02 08:41:16.492241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.492214 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.547883 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.547943 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.547987 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnf9l\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.548074 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.548214 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.548266 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.548358 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.548384 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.550106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.548447 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.563079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.561561 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk"] Nov 02 08:41:16.606931 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.606876 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv"] Nov 02 08:41:16.649675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649625 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.649675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649681 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649700 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649722 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnf9l\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649781 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649804 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649845 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649869 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649909 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.649960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.649941 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650082 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650127 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69h2t\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650246 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650233 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650280 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650294 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.650507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650396 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.650880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650520 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.650880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650549 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.650880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650580 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.650880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650605 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.650880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.650749 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.651376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.651350 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.652611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.652587 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.653086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.653064 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.703777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.703734 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.704693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.704662 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnf9l\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l\") pod \"maas-default-gateway-openshift-default-769df7846c-kgvsk\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.752363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752323 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752379 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752411 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752453 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752501 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69h2t\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752538 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752573 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.752600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752601 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.753000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752637 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.753000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752854 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.753000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752893 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.753000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.752954 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.753272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.753001 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.753272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.753194 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.755155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.755119 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.755345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.755325 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.761269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.761251 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:16.805546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.805486 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.806079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.806050 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69h2t\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t\") pod \"maas-default-gateway-openshift-default-769df7846c-gb7jv\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:16.809670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:16.809592 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:17.194229 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.194117 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk"] Nov 02 08:41:17.231843 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:41:17.231811 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fd0ce5_7958_41dd_9ac2_1fd3bdf261ed.slice/crio-caec3e703f4ea8f564416507abbb592ba90f95828c74327299e07707b9cda23a WatchSource:0}: Error finding container caec3e703f4ea8f564416507abbb592ba90f95828c74327299e07707b9cda23a: Status 404 returned error can't find the container with id caec3e703f4ea8f564416507abbb592ba90f95828c74327299e07707b9cda23a Nov 02 08:41:17.235065 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.235030 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:17.235167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.235103 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:17.235167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.235163 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:17.243028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.243000 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv"] Nov 02 08:41:17.247531 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:41:17.247506 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4381ccf4_b9f8_407c_a053_1a6ac3a352fd.slice/crio-ea1fac619e93332928e47983a8114903c9b1ec2e22903fd313278b02b49f519b WatchSource:0}: Error finding container ea1fac619e93332928e47983a8114903c9b1ec2e22903fd313278b02b49f519b: Status 404 returned error can't find the container with id ea1fac619e93332928e47983a8114903c9b1ec2e22903fd313278b02b49f519b Nov 02 08:41:17.250805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.250765 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:17.250915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.250852 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:17.250915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:17.250898 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:18.079812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.079766 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" event={"ID":"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed","Type":"ContainerStarted","Data":"b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a"} Nov 02 08:41:18.080362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.079841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" event={"ID":"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed","Type":"ContainerStarted","Data":"caec3e703f4ea8f564416507abbb592ba90f95828c74327299e07707b9cda23a"} Nov 02 08:41:18.081790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.081755 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" event={"ID":"4381ccf4-b9f8-407c-a053-1a6ac3a352fd","Type":"ContainerStarted","Data":"daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e"} Nov 02 08:41:18.081964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.081815 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" event={"ID":"4381ccf4-b9f8-407c-a053-1a6ac3a352fd","Type":"ContainerStarted","Data":"ea1fac619e93332928e47983a8114903c9b1ec2e22903fd313278b02b49f519b"} Nov 02 08:41:18.361798 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.361688 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podStartSLOduration=2.361673312 podStartE2EDuration="2.361673312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:18.244612302 +0000 UTC m=+2125.335232724" watchObservedRunningTime="2026-11-02 08:41:18.361673312 +0000 UTC m=+2125.452293723" Nov 02 08:41:18.362766 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.362735 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podStartSLOduration=2.362725419 podStartE2EDuration="2.362725419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:18.360403132 +0000 UTC m=+2125.451023548" watchObservedRunningTime="2026-11-02 08:41:18.362725419 +0000 UTC m=+2125.453345835" Nov 02 08:41:18.762211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.762172 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:18.763757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.763730 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:18.763842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.763793 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:18.814125 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.814088 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:18.814508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.814479 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:18.814701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:18.814677 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:19.762472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:19.762429 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:19.762903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:19.762509 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:19.810090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:19.810048 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:19.810437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:19.810405 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:20.347269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.347231 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q"] Nov 02 08:41:20.353167 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.352809 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.366945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.366914 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.367186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.367162 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.367368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.367349 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.367496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.367482 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.367731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.367715 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.367975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.367878 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.368171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.368119 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblcg\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.368331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.368311 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.368468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.368451 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.408165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.407178 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d5a5a-openshift-default-dockercfg-k8cwq\"" Nov 02 08:41:20.507678 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.494222 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.508531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.508485 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.510645 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515309 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515435 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515550 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515636 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515740 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblcg\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515797 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.588224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.515868 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.589990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.589943 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.608288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.606455 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.647165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.642538 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.672164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.663690 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.672164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.670325 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.679165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.673569 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblcg\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.679165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.676242 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.740165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.734911 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy\") pod \"e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:20.768732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.768683 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:20.769266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.768744 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:20.773907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.773878 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q"] Nov 02 08:41:20.812485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.812447 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:20.817218 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.817171 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:20.967802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:20.967765 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:21.647938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.647912 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q"] Nov 02 08:41:21.650580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.650536 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:21.650693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.650631 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:21.650792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.650695 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:21.775378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.775341 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:21.783930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.775953 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:21.813935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.813864 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:21.814233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:21.814130 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:22.178199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.175383 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" event={"ID":"b174008c-b394-4f4a-8b42-e0afda9e5150","Type":"ContainerStarted","Data":"8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44"} Nov 02 08:41:22.178199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.175436 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" event={"ID":"b174008c-b394-4f4a-8b42-e0afda9e5150","Type":"ContainerStarted","Data":"1f09d46909a5f0455cdf775a375c4ee0a64793cdc10afef585c988562f10526d"} Nov 02 08:41:22.316416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.316290 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podStartSLOduration=2.316269134 podStartE2EDuration="2.316269134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:22.311113228 +0000 UTC m=+2129.401733648" watchObservedRunningTime="2026-11-02 08:41:22.316269134 +0000 UTC m=+2129.406889557" Nov 02 08:41:22.762511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.762475 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:22.762700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.762528 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:22.826164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.815323 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:22.826164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.815382 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:22.970003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.969956 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:22.976456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:22.976426 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:23.044928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.044679 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb"] Nov 02 08:41:23.049701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.049664 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9d77b6-b9thb"] Nov 02 08:41:23.153253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.153219 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d280a306-acf2-4138-96c1-2df3003424d1" path="/var/lib/kubelet/pods/d280a306-acf2-4138-96c1-2df3003424d1/volumes" Nov 02 08:41:23.205171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.204483 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:23.230163 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.229446 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:41:23.762350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.762309 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:23.762523 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.762379 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:23.816275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.816236 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:23.816562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:23.816534 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:24.761717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:24.761670 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 08:41:24.762206 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:24.761747 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 08:41:24.782217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:24.782173 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:41:24.810393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:24.810350 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 08:41:24.810609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:24.810436 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 08:41:24.924708 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:41:24.924649 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:41:26.803402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:26.803355 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kgvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:41:26.804255 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:26.804213 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:41:26.856334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:26.856284 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-gb7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:41:26.856562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:26.856373 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:41:27.122484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:27.122391 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:27.126634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:27.126601 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:27.128979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:27.128956 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:27.129248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:27.129231 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:27.133375 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:27.133356 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:41:27.133830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:27.133811 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:41:30.150153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:30.150086 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2"] Nov 02 08:41:30.188735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:30.188683 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-108cf9-c79q2"] Nov 02 08:41:31.092183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:31.092115 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cbfd47-5703-4091-9985-dd7e40a6d611" path="/var/lib/kubelet/pods/97cbfd47-5703-4091-9985-dd7e40a6d611/volumes" Nov 02 08:41:54.510843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:54.510800 5055 generic.go:356] "Generic (PLEG): container finished" podID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerID="379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8" exitCode=0 Nov 02 08:41:54.511326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:54.510833 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" event={"ID":"bd96a9ac-a932-4d9e-914e-475702f42dd4","Type":"ContainerDied","Data":"379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8"} Nov 02 08:41:54.511422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:54.511404 5055 scope.go:122] "RemoveContainer" containerID="379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8" Nov 02 08:41:54.513217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:54.513183 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:54.513325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:54.513282 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:54.513325 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:54.513315 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:41:54.921183 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:41:54.921077 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:41:55.352616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:55.352574 5055 scope.go:122] "RemoveContainer" containerID="0dad46638448ca57ff745b17f8f5baca96c7bf25f9bed0144faee99e4249a3cb" Nov 02 08:41:55.527127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:55.526936 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" event={"ID":"bd96a9ac-a932-4d9e-914e-475702f42dd4","Type":"ContainerStarted","Data":"d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb"} Nov 02 08:41:55.552232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:55.552187 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:41:55.552562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:55.552529 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 08:41:55.552696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:55.552584 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 08:41:56.551775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:56.551738 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 08:41:56.552244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:56.551810 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 08:41:57.551821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:57.551782 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 08:41:57.552272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:57.551833 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 08:41:58.554466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:58.554426 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 08:41:58.555095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:58.554605 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 08:41:59.515259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:59.515204 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7"] Nov 02 08:41:59.552619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:59.552574 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 08:41:59.552793 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:59.552661 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 08:41:59.631800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:41:59.631764 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k"] Nov 02 08:42:00.552477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:00.552442 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 08:42:00.552712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:00.552505 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 08:42:01.361165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.359280 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c"] Nov 02 08:42:01.389907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.389797 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.415376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.415347 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c"] Nov 02 08:42:01.458204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.458165 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:42:01.458457 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.458305 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:42:01.458534 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.458509 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:42:01.484824 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.484790 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:42:01.544387 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.544339 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.544586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.544520 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.544638 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.544606 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltfn\" (UniqueName: \"kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.556773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.556736 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:42:01.557185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.557128 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:42:01.557981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.557959 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:42:01.645915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.645821 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.646092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.645917 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.646092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.646025 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltfn\" (UniqueName: \"kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.646788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.646758 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.648526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.648505 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.713651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.713616 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltfn\" (UniqueName: \"kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn\") pod \"maas-api-revoke-sub-e5cd785f-dng9c\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:01.716216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:01.714706 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:02.112950 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:02.111606 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c"] Nov 02 08:42:02.114551 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:42:02.114520 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548dafa7_36af_4778_80bd_6b89eea80a29.slice/crio-4ce93d4ca8be965de00b866f9c4469d7073507781e0009c3f323d9ba24e644dd WatchSource:0}: Error finding container 4ce93d4ca8be965de00b866f9c4469d7073507781e0009c3f323d9ba24e644dd: Status 404 returned error can't find the container with id 4ce93d4ca8be965de00b866f9c4469d7073507781e0009c3f323d9ba24e644dd Nov 02 08:42:02.587677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:02.587632 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" event={"ID":"548dafa7-36af-4778-80bd-6b89eea80a29","Type":"ContainerStarted","Data":"b33f6581719d6ed0ba84ccadf1ee72d5ff4ee787f5fcfc758fbf6b8335d9fd55"} Nov 02 08:42:02.588103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:02.587710 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" event={"ID":"548dafa7-36af-4778-80bd-6b89eea80a29","Type":"ContainerStarted","Data":"4ce93d4ca8be965de00b866f9c4469d7073507781e0009c3f323d9ba24e644dd"} Nov 02 08:42:02.695657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:02.695194 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" podStartSLOduration=1.695174234 podStartE2EDuration="1.695174234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:02.693248325 +0000 UTC m=+2169.783868745" watchObservedRunningTime="2026-11-02 08:42:02.695174234 +0000 UTC m=+2169.785794657" Nov 02 08:42:03.596886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:03.596848 5055 generic.go:356] "Generic (PLEG): container finished" podID="548dafa7-36af-4778-80bd-6b89eea80a29" containerID="b33f6581719d6ed0ba84ccadf1ee72d5ff4ee787f5fcfc758fbf6b8335d9fd55" exitCode=0 Nov 02 08:42:03.597345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:03.596887 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" event={"ID":"548dafa7-36af-4778-80bd-6b89eea80a29","Type":"ContainerDied","Data":"b33f6581719d6ed0ba84ccadf1ee72d5ff4ee787f5fcfc758fbf6b8335d9fd55"} Nov 02 08:42:04.516249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.516117 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" podUID="c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" containerName="payload-processing" containerID="cri-o://d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379" gracePeriod=25 Nov 02 08:42:04.632698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.632653 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" podUID="c5891317-7be7-4998-a815-f7af30b11c2a" containerName="payload-pre-processing" containerID="cri-o://6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb" gracePeriod=25 Nov 02 08:42:04.844901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.844861 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:04.880222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.880188 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:42:04.957332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.957298 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:42:04.992294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.992234 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca\") pod \"548dafa7-36af-4778-80bd-6b89eea80a29\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " Nov 02 08:42:04.992537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.992352 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq\" (UniqueName: \"kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq\") pod \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " Nov 02 08:42:04.992964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.992933 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca\") pod \"548dafa7-36af-4778-80bd-6b89eea80a29\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " Nov 02 08:42:04.993116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.992981 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume\") pod \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\" (UID: \"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c\") " Nov 02 08:42:04.993116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.993012 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn\" (UniqueName: \"kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn\") pod \"548dafa7-36af-4778-80bd-6b89eea80a29\" (UID: \"548dafa7-36af-4778-80bd-6b89eea80a29\") " Nov 02 08:42:04.993745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.993692 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca" pod "548dafa7-36af-4778-80bd-6b89eea80a29" (UID: "548dafa7-36af-4778-80bd-6b89eea80a29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:04.993745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.993708 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume" pod "c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" (UID: "c43ffc68-8bcc-4851-a087-3ed4a0e0e36c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:04.994300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.994275 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:04.994415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.994303 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:04.996972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.996930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca" pod "548dafa7-36af-4778-80bd-6b89eea80a29" (UID: "548dafa7-36af-4778-80bd-6b89eea80a29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:04.997186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.997127 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq" pod "c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" (UID: "c43ffc68-8bcc-4851-a087-3ed4a0e0e36c"). InnerVolumeSpecName "kube-api-access-tbrkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:04.998974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:04.998943 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn" pod "548dafa7-36af-4778-80bd-6b89eea80a29" (UID: "548dafa7-36af-4778-80bd-6b89eea80a29"). InnerVolumeSpecName "kube-api-access-9ltfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:05.095055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.094952 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc\" (UniqueName: \"kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc\") pod \"c5891317-7be7-4998-a815-f7af30b11c2a\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " Nov 02 08:42:05.095305 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.095120 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume\") pod \"c5891317-7be7-4998-a815-f7af30b11c2a\" (UID: \"c5891317-7be7-4998-a815-f7af30b11c2a\") " Nov 02 08:42:05.095914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.095882 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ltfn\" (UniqueName: \"kubernetes.io/projected/548dafa7-36af-4778-80bd-6b89eea80a29-kube-api-access-9ltfn\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:05.095914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.095916 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548dafa7-36af-4778-80bd-6b89eea80a29-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:05.096129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.095932 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbrkq\" (UniqueName: \"kubernetes.io/projected/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c-kube-api-access-tbrkq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:05.096129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.095933 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume" pod "c5891317-7be7-4998-a815-f7af30b11c2a" (UID: "c5891317-7be7-4998-a815-f7af30b11c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:05.097613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.097580 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc" pod "c5891317-7be7-4998-a815-f7af30b11c2a" (UID: "c5891317-7be7-4998-a815-f7af30b11c2a"). InnerVolumeSpecName "kube-api-access-rdcrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:05.197588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.197413 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcrc\" (UniqueName: \"kubernetes.io/projected/c5891317-7be7-4998-a815-f7af30b11c2a-kube-api-access-rdcrc\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:05.197588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.197456 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5891317-7be7-4998-a815-f7af30b11c2a-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:05.625748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.625712 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" Nov 02 08:42:05.625914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.625749 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c" event={"ID":"548dafa7-36af-4778-80bd-6b89eea80a29","Type":"ContainerDied","Data":"4ce93d4ca8be965de00b866f9c4469d7073507781e0009c3f323d9ba24e644dd"} Nov 02 08:42:05.625914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.625833 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ce93d4ca8be965de00b866f9c4469d7073507781e0009c3f323d9ba24e644dd" Nov 02 08:42:05.627347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.627307 5055 generic.go:356] "Generic (PLEG): container finished" podID="c5891317-7be7-4998-a815-f7af30b11c2a" containerID="6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb" exitCode=0 Nov 02 08:42:05.627527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.627361 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" Nov 02 08:42:05.627711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.627391 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" event={"ID":"c5891317-7be7-4998-a815-f7af30b11c2a","Type":"ContainerDied","Data":"6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb"} Nov 02 08:42:05.627843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.627740 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k" event={"ID":"c5891317-7be7-4998-a815-f7af30b11c2a","Type":"ContainerDied","Data":"5d98f10f053f4e778aaa973b6d962686f1a7e042cf98e3b932b18b9dbea5f4f1"} Nov 02 08:42:05.627843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.627775 5055 scope.go:122] "RemoveContainer" containerID="6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb" Nov 02 08:42:05.629252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.629031 5055 generic.go:356] "Generic (PLEG): container finished" podID="c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" containerID="d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379" exitCode=0 Nov 02 08:42:05.629252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.629060 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" event={"ID":"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c","Type":"ContainerDied","Data":"d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379"} Nov 02 08:42:05.629252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.629091 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" Nov 02 08:42:05.629252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.629094 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7" event={"ID":"c43ffc68-8bcc-4851-a087-3ed4a0e0e36c","Type":"ContainerDied","Data":"b3462d7962b227e076ff05523b64a65dc1295f5f254313d6c3d7def37c9e0264"} Nov 02 08:42:05.639440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.639419 5055 scope.go:122] "RemoveContainer" containerID="6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb" Nov 02 08:42:05.639735 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:05.639715 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb\": container with ID starting with 6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb not found: ID does not exist" containerID="6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb" Nov 02 08:42:05.639782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.639743 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb"} err="failed to get container status \"6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb\": rpc error: code = NotFound desc = could not find container \"6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb\": container with ID starting with 6413312cde3d508807837f31c9c4b488afd173215f66984c9ce2ebba872e76cb not found: ID does not exist" Nov 02 08:42:05.639782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.639762 5055 scope.go:122] "RemoveContainer" containerID="d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379" Nov 02 08:42:05.649422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.649404 5055 scope.go:122] "RemoveContainer" containerID="d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379" Nov 02 08:42:05.649693 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:05.649671 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379\": container with ID starting with d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379 not found: ID does not exist" containerID="d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379" Nov 02 08:42:05.649786 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.649698 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379"} err="failed to get container status \"d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379\": rpc error: code = NotFound desc = could not find container \"d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379\": container with ID starting with d375369fc8de20ac991624f65afdd8cca4a26fb6dae237057be4c2d199a8d379 not found: ID does not exist" Nov 02 08:42:05.830026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.829936 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k"] Nov 02 08:42:05.833821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.833785 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-85875869cf-srd4k"] Nov 02 08:42:05.932732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.932687 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7"] Nov 02 08:42:05.936390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:05.936359 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceb40-59dfcd476c-qzzh7"] Nov 02 08:42:07.122579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:07.120343 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" path="/var/lib/kubelet/pods/c43ffc68-8bcc-4851-a087-3ed4a0e0e36c/volumes" Nov 02 08:42:07.122579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:07.121109 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5891317-7be7-4998-a815-f7af30b11c2a" path="/var/lib/kubelet/pods/c5891317-7be7-4998-a815-f7af30b11c2a/volumes" Nov 02 08:42:08.283577 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:08.283544 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s"] Nov 02 08:42:08.329858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:08.329820 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45fc31-tk87s"] Nov 02 08:42:09.102443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:09.102408 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6cdac9-cbb9-4fa6-8c06-817564ee9be5" path="/var/lib/kubelet/pods/2c6cdac9-cbb9-4fa6-8c06-817564ee9be5/volumes" Nov 02 08:42:14.742593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:14.741478 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2"] Nov 02 08:42:14.742593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:14.741854 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" containerID="cri-o://df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618" gracePeriod=30 Nov 02 08:42:15.620866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.620830 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq"] Nov 02 08:42:15.621419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621405 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" containerName="payload-processing" Nov 02 08:42:15.621467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621422 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" containerName="payload-processing" Nov 02 08:42:15.621467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621441 5055 cpu_manager.go:414] "containerMap: removing container" podUID="548dafa7-36af-4778-80bd-6b89eea80a29" containerName="revoke-keys" Nov 02 08:42:15.621467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621446 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548dafa7-36af-4778-80bd-6b89eea80a29" containerName="revoke-keys" Nov 02 08:42:15.621467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621463 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c5891317-7be7-4998-a815-f7af30b11c2a" containerName="payload-pre-processing" Nov 02 08:42:15.621602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621468 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5891317-7be7-4998-a815-f7af30b11c2a" containerName="payload-pre-processing" Nov 02 08:42:15.621602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621575 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43ffc68-8bcc-4851-a087-3ed4a0e0e36c" containerName="payload-processing" Nov 02 08:42:15.621602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621586 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="548dafa7-36af-4778-80bd-6b89eea80a29" containerName="revoke-keys" Nov 02 08:42:15.621602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.621595 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5891317-7be7-4998-a815-f7af30b11c2a" containerName="payload-pre-processing" Nov 02 08:42:15.627008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.626982 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.675640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.675581 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:42:15.675799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.675678 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:42:15.721765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.721709 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:42:15.721979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.721777 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:42:15.726934 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.726899 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq"] Nov 02 08:42:15.755366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.755336 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvlz\" (UniqueName: \"kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.755755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.755403 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.755755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.755468 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.856854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.856815 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.856854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.856857 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.857084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.857065 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvlz\" (UniqueName: \"kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.857552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.857525 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.859388 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.859359 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.911861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.911782 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvlz\" (UniqueName: \"kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:15.941431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:15.941392 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:16.324180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:16.324043 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq"] Nov 02 08:42:16.353456 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:42:16.353416 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54392f8b_b6a0_4e47_ad9c_a85499485a63.slice/crio-86186ce488b85e1388077b779e83fc4dbfba83478691f7d416e73a906d132be1 WatchSource:0}: Error finding container 86186ce488b85e1388077b779e83fc4dbfba83478691f7d416e73a906d132be1: Status 404 returned error can't find the container with id 86186ce488b85e1388077b779e83fc4dbfba83478691f7d416e73a906d132be1 Nov 02 08:42:16.479465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:16.479426 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:42:16.765296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:16.765250 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" event={"ID":"54392f8b-b6a0-4e47-ad9c-a85499485a63","Type":"ContainerStarted","Data":"86186ce488b85e1388077b779e83fc4dbfba83478691f7d416e73a906d132be1"} Nov 02 08:42:17.774577 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:17.774483 5055 generic.go:356] "Generic (PLEG): container finished" podID="54392f8b-b6a0-4e47-ad9c-a85499485a63" containerID="13284d34f711246f447c4b4341cbbd3a556c170f1c90afad6c998ce0401ccac2" exitCode=0 Nov 02 08:42:17.774958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:17.774573 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" event={"ID":"54392f8b-b6a0-4e47-ad9c-a85499485a63","Type":"ContainerDied","Data":"13284d34f711246f447c4b4341cbbd3a556c170f1c90afad6c998ce0401ccac2"} Nov 02 08:42:19.012780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.012515 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:19.130897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.130548 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz\" (UniqueName: \"kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz\") pod \"54392f8b-b6a0-4e47-ad9c-a85499485a63\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " Nov 02 08:42:19.130897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.130628 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca\") pod \"54392f8b-b6a0-4e47-ad9c-a85499485a63\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " Nov 02 08:42:19.131748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.131703 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca" pod "54392f8b-b6a0-4e47-ad9c-a85499485a63" (UID: "54392f8b-b6a0-4e47-ad9c-a85499485a63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:19.132023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.131964 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca\") pod \"54392f8b-b6a0-4e47-ad9c-a85499485a63\" (UID: \"54392f8b-b6a0-4e47-ad9c-a85499485a63\") " Nov 02 08:42:19.132771 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.132736 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:19.134959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.134932 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz" pod "54392f8b-b6a0-4e47-ad9c-a85499485a63" (UID: "54392f8b-b6a0-4e47-ad9c-a85499485a63"). InnerVolumeSpecName "kube-api-access-mcvlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:19.136519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.136476 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca" pod "54392f8b-b6a0-4e47-ad9c-a85499485a63" (UID: "54392f8b-b6a0-4e47-ad9c-a85499485a63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:19.234400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.234367 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcvlz\" (UniqueName: \"kubernetes.io/projected/54392f8b-b6a0-4e47-ad9c-a85499485a63-kube-api-access-mcvlz\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:19.234400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.234398 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54392f8b-b6a0-4e47-ad9c-a85499485a63-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:19.816028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.815987 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" event={"ID":"54392f8b-b6a0-4e47-ad9c-a85499485a63","Type":"ContainerDied","Data":"86186ce488b85e1388077b779e83fc4dbfba83478691f7d416e73a906d132be1"} Nov 02 08:42:19.816233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.816043 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86186ce488b85e1388077b779e83fc4dbfba83478691f7d416e73a906d132be1" Nov 02 08:42:19.816233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:19.816106 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq" Nov 02 08:42:20.126885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.126857 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:42:20.208336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.208295 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw"] Nov 02 08:42:20.248108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248054 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248179 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248255 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248348 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248377 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248430 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248513 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248478 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248524 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:20.248617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248577 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.248675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.248628 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket\") pod \"108ddb47-d760-4bc5-97cc-25716febd86a\" (UID: \"108ddb47-d760-4bc5-97cc-25716febd86a\") " Nov 02 08:42:20.249386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.249039 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:20.249386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.249055 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:20.249386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.249248 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ddb47-d760-4bc5-97cc-25716febd86a-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.249386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.249272 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.249386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.249289 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.249386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.249356 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:20.250153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.250108 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:20.252809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.252347 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "kube-api-access-shbx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:20.253198 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.253177 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:20.253289 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.253262 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:20.253361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.253345 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo" pod "108ddb47-d760-4bc5-97cc-25716febd86a" (UID: "108ddb47-d760-4bc5-97cc-25716febd86a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:42:20.355008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.354653 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.355008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.354716 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ddb47-d760-4bc5-97cc-25716febd86a-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.355008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.354727 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.355008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.354736 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ddb47-d760-4bc5-97cc-25716febd86a-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.355008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.354746 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shbx9\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-kube-api-access-shbx9\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.355008 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.354756 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ddb47-d760-4bc5-97cc-25716febd86a-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:20.828559 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.828518 5055 generic.go:356] "Generic (PLEG): container finished" podID="108ddb47-d760-4bc5-97cc-25716febd86a" containerID="df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618" exitCode=0 Nov 02 08:42:20.828788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.828600 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" Nov 02 08:42:20.828788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.828599 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" event={"ID":"108ddb47-d760-4bc5-97cc-25716febd86a","Type":"ContainerDied","Data":"df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618"} Nov 02 08:42:20.828788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.828710 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2" event={"ID":"108ddb47-d760-4bc5-97cc-25716febd86a","Type":"ContainerDied","Data":"d0bc79f1b7d0b3592ccd26856ba4139d5f1a6094467bd5f72a6770665bc5520a"} Nov 02 08:42:20.828788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.828743 5055 scope.go:122] "RemoveContainer" containerID="df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618" Nov 02 08:42:20.840842 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.840818 5055 scope.go:122] "RemoveContainer" containerID="ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36" Nov 02 08:42:20.851492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.851466 5055 scope.go:122] "RemoveContainer" containerID="df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618" Nov 02 08:42:20.851795 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:20.851772 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618\": container with ID starting with df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618 not found: ID does not exist" containerID="df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618" Nov 02 08:42:20.851871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.851806 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618"} err="failed to get container status \"df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618\": rpc error: code = NotFound desc = could not find container \"df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618\": container with ID starting with df217cdbb3694d0f3990f14e8aa07e318934c973c19de7c5def865567abf1618 not found: ID does not exist" Nov 02 08:42:20.851871 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.851826 5055 scope.go:122] "RemoveContainer" containerID="ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36" Nov 02 08:42:20.852105 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:20.852086 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36\": container with ID starting with ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36 not found: ID does not exist" containerID="ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36" Nov 02 08:42:20.852161 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:20.852111 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36"} err="failed to get container status \"ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36\": rpc error: code = NotFound desc = could not find container \"ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36\": container with ID starting with ab965dc054787a80ccf59c527a3fa2653f5139a32a6e0e8f20db513c08f9be36 not found: ID does not exist" Nov 02 08:42:21.009164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:21.003767 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2"] Nov 02 08:42:21.030651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:21.029982 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45fc31-openshift-default-9658b4cf7-n6hv2"] Nov 02 08:42:21.122064 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:21.121812 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" path="/var/lib/kubelet/pods/108ddb47-d760-4bc5-97cc-25716febd86a/volumes" Nov 02 08:42:24.927632 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:24.927590 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:42:26.896045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:26.895958 5055 generic.go:356] "Generic (PLEG): container finished" podID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerID="d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c" exitCode=0 Nov 02 08:42:26.896045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:26.896029 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" event={"ID":"695546c1-9771-4a9b-a2bd-23ef70aa73f5","Type":"ContainerDied","Data":"d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c"} Nov 02 08:42:26.896509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:26.896480 5055 scope.go:122] "RemoveContainer" containerID="d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c" Nov 02 08:42:26.898062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:26.898024 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:42:26.898173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:26.898100 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:42:26.898173 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:26.898131 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:42:27.936822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:27.936768 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" event={"ID":"695546c1-9771-4a9b-a2bd-23ef70aa73f5","Type":"ContainerStarted","Data":"581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0"} Nov 02 08:42:28.449454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:28.449417 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:42:28.450717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:28.450687 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 08:42:28.450881 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:28.450739 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 08:42:29.450030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:29.449988 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 08:42:29.450546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:29.450066 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 08:42:30.450304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:30.450267 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 08:42:30.450728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:30.450327 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 08:42:30.823002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:30.822903 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:42:31.450105 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:31.450065 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 08:42:31.450356 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:31.450120 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 08:42:32.450468 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:32.450423 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 08:42:32.450889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:32.450488 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 08:42:33.455527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:33.455494 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:42:33.456657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:33.456628 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:42:35.230734 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.230690 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" containerID="cri-o://68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b" gracePeriod=45 Nov 02 08:42:35.535761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.535725 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:42:35.659551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.659514 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache\") pod \"c9af39d1-64a6-4c73-a63b-32b614bba48f\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " Nov 02 08:42:35.659758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.659650 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home\") pod \"c9af39d1-64a6-4c73-a63b-32b614bba48f\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " Nov 02 08:42:35.659835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.659759 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir\") pod \"c9af39d1-64a6-4c73-a63b-32b614bba48f\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " Nov 02 08:42:35.659835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.659809 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp\" (UniqueName: \"kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp\") pod \"c9af39d1-64a6-4c73-a63b-32b614bba48f\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " Nov 02 08:42:35.659955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.659924 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs\") pod \"c9af39d1-64a6-4c73-a63b-32b614bba48f\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " Nov 02 08:42:35.660019 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.659965 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm\") pod \"c9af39d1-64a6-4c73-a63b-32b614bba48f\" (UID: \"c9af39d1-64a6-4c73-a63b-32b614bba48f\") " Nov 02 08:42:35.660507 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.660478 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir" pod "c9af39d1-64a6-4c73-a63b-32b614bba48f" (UID: "c9af39d1-64a6-4c73-a63b-32b614bba48f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.660841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.660810 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache" pod "c9af39d1-64a6-4c73-a63b-32b614bba48f" (UID: "c9af39d1-64a6-4c73-a63b-32b614bba48f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.660955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.660879 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home" pod "c9af39d1-64a6-4c73-a63b-32b614bba48f" (UID: "c9af39d1-64a6-4c73-a63b-32b614bba48f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.663369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.663324 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs" pod "c9af39d1-64a6-4c73-a63b-32b614bba48f" (UID: "c9af39d1-64a6-4c73-a63b-32b614bba48f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:35.663482 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.663430 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm" pod "c9af39d1-64a6-4c73-a63b-32b614bba48f" (UID: "c9af39d1-64a6-4c73-a63b-32b614bba48f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.663552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.663478 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp" pod "c9af39d1-64a6-4c73-a63b-32b614bba48f" (UID: "c9af39d1-64a6-4c73-a63b-32b614bba48f"). InnerVolumeSpecName "kube-api-access-knssp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:35.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.762058 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.762099 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knssp\" (UniqueName: \"kubernetes.io/projected/c9af39d1-64a6-4c73-a63b-32b614bba48f-kube-api-access-knssp\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.762114 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9af39d1-64a6-4c73-a63b-32b614bba48f-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.762129 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.762175 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.762199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:35.762188 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9af39d1-64a6-4c73-a63b-32b614bba48f-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:36.039394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.039291 5055 generic.go:356] "Generic (PLEG): container finished" podID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerID="68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b" exitCode=0 Nov 02 08:42:36.039394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.039350 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" Nov 02 08:42:36.039633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.039353 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" event={"ID":"c9af39d1-64a6-4c73-a63b-32b614bba48f","Type":"ContainerDied","Data":"68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b"} Nov 02 08:42:36.039633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.039460 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw" event={"ID":"c9af39d1-64a6-4c73-a63b-32b614bba48f","Type":"ContainerDied","Data":"9842cc6f253e1fd2ca8c06f0531b0d282acec5b892199e6242d953f58aac37b2"} Nov 02 08:42:36.039633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.039477 5055 scope.go:122] "RemoveContainer" containerID="68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b" Nov 02 08:42:36.050997 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.050968 5055 scope.go:122] "RemoveContainer" containerID="68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b" Nov 02 08:42:36.051339 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:36.051317 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b\": container with ID starting with 68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b not found: ID does not exist" containerID="68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b" Nov 02 08:42:36.051426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.051351 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b"} err="failed to get container status \"68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b\": rpc error: code = NotFound desc = could not find container \"68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b\": container with ID starting with 68a14474668d23d0627c28593e21819b0a2e20be8b3a67346dceddaa66e9f03b not found: ID does not exist" Nov 02 08:42:36.197358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.197320 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw"] Nov 02 08:42:36.245041 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:36.244999 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/premium-w4-8e6407-kserve-6b58695b6-57fnw"] Nov 02 08:42:37.094272 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:37.094228 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" path="/var/lib/kubelet/pods/c9af39d1-64a6-4c73-a63b-32b614bba48f/volumes" Nov 02 08:42:43.771804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:43.771766 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:42:53.614557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.614512 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp"] Nov 02 08:42:53.615538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615506 5055 cpu_manager.go:414] "containerMap: removing container" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615539 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615557 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615565 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615576 5055 cpu_manager.go:414] "containerMap: removing container" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615583 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615610 5055 cpu_manager.go:414] "containerMap: removing container" podUID="54392f8b-b6a0-4e47-ad9c-a85499485a63" containerName="revoke-keys" Nov 02 08:42:53.615704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615616 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54392f8b-b6a0-4e47-ad9c-a85499485a63" containerName="revoke-keys" Nov 02 08:42:53.616067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615809 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" Nov 02 08:42:53.616067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615824 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9af39d1-64a6-4c73-a63b-32b614bba48f" containerName="main" Nov 02 08:42:53.616067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615837 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="108ddb47-d760-4bc5-97cc-25716febd86a" containerName="istio-proxy" Nov 02 08:42:53.616067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.615852 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="54392f8b-b6a0-4e47-ad9c-a85499485a63" containerName="revoke-keys" Nov 02 08:42:53.620040 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.620014 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.668582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.668534 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4"] Nov 02 08:42:53.675393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.675355 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.708735 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.708687 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:42:53.709267 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.709242 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:42:53.709681 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.709662 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:42:53.710353 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.710279 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:42:53.721830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.721790 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp"] Nov 02 08:42:53.732162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.731819 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4"] Nov 02 08:42:53.739302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.739265 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.739636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.739610 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.739823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.739804 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.740085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.740062 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wk8z\" (UniqueName: \"kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.740319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.740300 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxd7z\" (UniqueName: \"kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.740456 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.740440 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.842099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.842295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842111 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk8z\" (UniqueName: \"kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.842295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842185 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxd7z\" (UniqueName: \"kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.842295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842214 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.842532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842320 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.842532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842437 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.842928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.842892 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.843112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.843088 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.844877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.844850 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.845347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.845326 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.895276 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.895194 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wk8z\" (UniqueName: \"kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z\") pod \"maas-api-revoke-sub-ac608733-xpntp\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:53.895448 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.895287 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxd7z\" (UniqueName: \"kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z\") pod \"maas-api-revoke-sub-471ca7bd-xbjp4\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:53.936086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.936047 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:54.000018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:53.999975 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:54.246284 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:54.246247 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:42:54.246478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:54.246307 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:42:54.322727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:54.322699 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp"] Nov 02 08:42:54.323753 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:42:54.323714 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf4bce4_c1d2_4615_b5ab_8aa91eae66f8.slice/crio-e7f14b03c58d536cb579507e8c3d08cbe20ab9f59aec027a69080a9815f46c2c WatchSource:0}: Error finding container e7f14b03c58d536cb579507e8c3d08cbe20ab9f59aec027a69080a9815f46c2c: Status 404 returned error can't find the container with id e7f14b03c58d536cb579507e8c3d08cbe20ab9f59aec027a69080a9815f46c2c Nov 02 08:42:54.417266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:54.417234 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4"] Nov 02 08:42:54.460303 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:42:54.460269 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bca7db_8bfd_40a7_97c8_823dcb58ffda.slice/crio-83c3ae3fac0eebba60c1dd1ba643aeebb2536be0aa1ee3573ed13a62c06c900f WatchSource:0}: Error finding container 83c3ae3fac0eebba60c1dd1ba643aeebb2536be0aa1ee3573ed13a62c06c900f: Status 404 returned error can't find the container with id 83c3ae3fac0eebba60c1dd1ba643aeebb2536be0aa1ee3573ed13a62c06c900f Nov 02 08:42:54.922197 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:42:54.922151 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:42:55.294671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.294624 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" event={"ID":"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8","Type":"ContainerStarted","Data":"7e9f77faf5c0dea560e951401ec990fe014c1448e234ba7707995d9e44d7acc8"} Nov 02 08:42:55.294938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.294700 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" event={"ID":"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8","Type":"ContainerStarted","Data":"e7f14b03c58d536cb579507e8c3d08cbe20ab9f59aec027a69080a9815f46c2c"} Nov 02 08:42:55.296612 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.296574 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" event={"ID":"61bca7db-8bfd-40a7-97c8-823dcb58ffda","Type":"ContainerStarted","Data":"020a10c098be9ae2ec7007fb7d63658651187b806e33392d02e45ae77caa8d63"} Nov 02 08:42:55.296778 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.296627 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" event={"ID":"61bca7db-8bfd-40a7-97c8-823dcb58ffda","Type":"ContainerStarted","Data":"83c3ae3fac0eebba60c1dd1ba643aeebb2536be0aa1ee3573ed13a62c06c900f"} Nov 02 08:42:55.298499 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.298470 5055 generic.go:356] "Generic (PLEG): container finished" podID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerID="8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44" exitCode=0 Nov 02 08:42:55.298646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.298539 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" event={"ID":"b174008c-b394-4f4a-8b42-e0afda9e5150","Type":"ContainerDied","Data":"8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44"} Nov 02 08:42:55.298964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.298946 5055 scope.go:122] "RemoveContainer" containerID="8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44" Nov 02 08:42:55.300761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.300724 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:42:55.300866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.300815 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:42:55.300926 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.300864 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:42:55.400376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.400321 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" podStartSLOduration=2.400303842 podStartE2EDuration="2.400303842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:55.399965423 +0000 UTC m=+2222.490585846" watchObservedRunningTime="2026-11-02 08:42:55.400303842 +0000 UTC m=+2222.490924259" Nov 02 08:42:55.425155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.425101 5055 scope.go:122] "RemoveContainer" containerID="530ccce5d61ef1d9372df18ebc8f67ed3237af00f12464b3188b10b3dbe7b919" Nov 02 08:42:55.543678 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.543646 5055 scope.go:122] "RemoveContainer" containerID="a919a391ca022b682286eab734e12fa2f14f11d73f4a285c4fec86128029ca9a" Nov 02 08:42:55.558193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.558165 5055 scope.go:122] "RemoveContainer" containerID="21c2dc59393c69d8f0d69fb49da6c42336bbad2184c340128ea60c76ea1b1822" Nov 02 08:42:55.577550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.577525 5055 scope.go:122] "RemoveContainer" containerID="86aed61b14f1a250385ab5540c7b04a184d8f3fd8bba2ed2ed1e23aaa6564499" Nov 02 08:42:55.633174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:55.632498 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" podStartSLOduration=2.632471427 podStartE2EDuration="2.632471427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:55.618315015 +0000 UTC m=+2222.708935434" watchObservedRunningTime="2026-11-02 08:42:55.632471427 +0000 UTC m=+2222.723091847" Nov 02 08:42:56.311306 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.311267 5055 generic.go:356] "Generic (PLEG): container finished" podID="1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" containerID="7e9f77faf5c0dea560e951401ec990fe014c1448e234ba7707995d9e44d7acc8" exitCode=0 Nov 02 08:42:56.311853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.311353 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" event={"ID":"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8","Type":"ContainerDied","Data":"7e9f77faf5c0dea560e951401ec990fe014c1448e234ba7707995d9e44d7acc8"} Nov 02 08:42:56.313539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.313506 5055 generic.go:356] "Generic (PLEG): container finished" podID="61bca7db-8bfd-40a7-97c8-823dcb58ffda" containerID="020a10c098be9ae2ec7007fb7d63658651187b806e33392d02e45ae77caa8d63" exitCode=0 Nov 02 08:42:56.313733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.313596 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" event={"ID":"61bca7db-8bfd-40a7-97c8-823dcb58ffda","Type":"ContainerDied","Data":"020a10c098be9ae2ec7007fb7d63658651187b806e33392d02e45ae77caa8d63"} Nov 02 08:42:56.316249 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.316203 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" event={"ID":"b174008c-b394-4f4a-8b42-e0afda9e5150","Type":"ContainerStarted","Data":"27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f"} Nov 02 08:42:56.968446 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.968411 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:42:56.968910 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.968886 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 08:42:56.968999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:56.968937 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 08:42:57.667710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.667684 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:57.670827 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.670806 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:57.767112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767075 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z\" (UniqueName: \"kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z\") pod \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " Nov 02 08:42:57.767297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767128 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca\") pod \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " Nov 02 08:42:57.767297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767192 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca\") pod \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " Nov 02 08:42:57.767297 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767289 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca\") pod \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\" (UID: \"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8\") " Nov 02 08:42:57.767406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767379 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca\") pod \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " Nov 02 08:42:57.767466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767409 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z\" (UniqueName: \"kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z\") pod \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\" (UID: \"61bca7db-8bfd-40a7-97c8-823dcb58ffda\") " Nov 02 08:42:57.767660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767634 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca" pod "61bca7db-8bfd-40a7-97c8-823dcb58ffda" (UID: "61bca7db-8bfd-40a7-97c8-823dcb58ffda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:57.767751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767666 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca" pod "1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" (UID: "1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:57.767922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767891 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:57.767922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.767917 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:57.769410 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.769388 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z" pod "1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" (UID: "1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8"). InnerVolumeSpecName "kube-api-access-9wk8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:57.769589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.769569 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z" pod "61bca7db-8bfd-40a7-97c8-823dcb58ffda" (UID: "61bca7db-8bfd-40a7-97c8-823dcb58ffda"). InnerVolumeSpecName "kube-api-access-qxd7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:57.769909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.769887 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca" pod "1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" (UID: "1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:57.770005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.769889 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca" pod "61bca7db-8bfd-40a7-97c8-823dcb58ffda" (UID: "61bca7db-8bfd-40a7-97c8-823dcb58ffda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:57.869639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.869546 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bca7db-8bfd-40a7-97c8-823dcb58ffda-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:57.869639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.869587 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxd7z\" (UniqueName: \"kubernetes.io/projected/61bca7db-8bfd-40a7-97c8-823dcb58ffda-kube-api-access-qxd7z\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:57.869639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.869604 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wk8z\" (UniqueName: \"kubernetes.io/projected/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-kube-api-access-9wk8z\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:57.869639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.869620 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:57.969121 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.969082 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 08:42:57.969333 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:57.969136 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 08:42:58.335340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.335247 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" Nov 02 08:42:58.335340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.335263 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp" event={"ID":"1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8","Type":"ContainerDied","Data":"e7f14b03c58d536cb579507e8c3d08cbe20ab9f59aec027a69080a9815f46c2c"} Nov 02 08:42:58.335340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.335317 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f14b03c58d536cb579507e8c3d08cbe20ab9f59aec027a69080a9815f46c2c" Nov 02 08:42:58.337030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.336997 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" event={"ID":"61bca7db-8bfd-40a7-97c8-823dcb58ffda","Type":"ContainerDied","Data":"83c3ae3fac0eebba60c1dd1ba643aeebb2536be0aa1ee3573ed13a62c06c900f"} Nov 02 08:42:58.337179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.337055 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c3ae3fac0eebba60c1dd1ba643aeebb2536be0aa1ee3573ed13a62c06c900f" Nov 02 08:42:58.337179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.337120 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4" Nov 02 08:42:58.972508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.972408 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 08:42:58.972508 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:58.972481 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 08:42:59.968738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:59.968698 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 08:42:59.968930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:42:59.968761 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 08:43:00.968922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:00.968887 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 08:43:00.969472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:00.968953 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 08:43:01.969001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:01.968947 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 08:43:01.969458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:01.969030 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 08:43:02.955994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.955954 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2"] Nov 02 08:43:02.956565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.956549 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" containerName="revoke-keys" Nov 02 08:43:02.956648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.956566 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" containerName="revoke-keys" Nov 02 08:43:02.956648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.956582 5055 cpu_manager.go:414] "containerMap: removing container" podUID="61bca7db-8bfd-40a7-97c8-823dcb58ffda" containerName="revoke-keys" Nov 02 08:43:02.956648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.956587 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bca7db-8bfd-40a7-97c8-823dcb58ffda" containerName="revoke-keys" Nov 02 08:43:02.956755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.956717 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" containerName="revoke-keys" Nov 02 08:43:02.956755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.956726 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bca7db-8bfd-40a7-97c8-823dcb58ffda" containerName="revoke-keys" Nov 02 08:43:02.960607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.960581 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:02.972844 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.972814 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:43:02.975149 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.975114 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:43:02.975887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:02.975866 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:43:03.013835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.013802 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d91f6f-openshift-default-dockercfg-tfgnb\"" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.067916 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.067970 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068053 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068171 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068231 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzgg\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068351 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068380 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068464 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.072165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.068498 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.089454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.089378 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2"] Nov 02 08:43:03.171155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.170105 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.171155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.170217 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzgg\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.171155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.170320 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.171155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.170351 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.171660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.171380 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.173352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.173324 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.173480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.173390 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.173480 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.173419 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.173608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.173517 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.174028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.173114 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.174227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.172801 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.174453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.174344 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.174633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.174611 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.175355 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.175334 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.175831 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.175809 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.176617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.176589 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.228693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.228607 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.228922 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.228844 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzgg\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg\") pod \"e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.275889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.275856 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:03.758890 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:43:03.758845 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058b922b_5389_4fc1_a032_1a49596e1f31.slice/crio-94daab2cede4386065b1b0b26168703ad3edb14acb86f7d1faaeffd0c56fe838 WatchSource:0}: Error finding container 94daab2cede4386065b1b0b26168703ad3edb14acb86f7d1faaeffd0c56fe838: Status 404 returned error can't find the container with id 94daab2cede4386065b1b0b26168703ad3edb14acb86f7d1faaeffd0c56fe838 Nov 02 08:43:03.760113 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.759798 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2"] Nov 02 08:43:03.762265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.762232 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:43:03.762386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.762309 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:43:03.762386 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:03.762355 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:43:04.396740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:04.396701 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" event={"ID":"058b922b-5389-4fc1-a032-1a49596e1f31","Type":"ContainerStarted","Data":"4d3a1fcc96bf01cb9e71941616f9f13cc285c1d6d2695b9e13085dabad67a3b9"} Nov 02 08:43:04.397322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:04.396761 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" event={"ID":"058b922b-5389-4fc1-a032-1a49596e1f31","Type":"ContainerStarted","Data":"94daab2cede4386065b1b0b26168703ad3edb14acb86f7d1faaeffd0c56fe838"} Nov 02 08:43:04.518385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:04.518315 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podStartSLOduration=2.51829631 podStartE2EDuration="2.51829631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:04.514914418 +0000 UTC m=+2231.605534838" watchObservedRunningTime="2026-11-02 08:43:04.51829631 +0000 UTC m=+2231.608916788" Nov 02 08:43:05.277052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:05.277008 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:05.286181 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:05.286118 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:05.405269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:05.405231 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:05.406244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:05.406225 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:43:09.797244 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.797134 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp"] Nov 02 08:43:09.802313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.802278 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:09.897465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.897427 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:43:09.897465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.897458 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:43:09.897704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.897475 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:43:09.897704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.897548 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:43:09.901498 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.901468 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp"] Nov 02 08:43:09.963773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.963728 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:09.963961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.963784 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxztm\" (UniqueName: \"kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:09.963961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:09.963856 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.065814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.065717 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.065814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.065765 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxztm\" (UniqueName: \"kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.065814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.065789 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.066665 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.066635 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.068402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.068378 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.119925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.119879 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxztm\" (UniqueName: \"kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.417737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.417637 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:10.852294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:10.852252 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp"] Nov 02 08:43:11.457787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:11.457745 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" event={"ID":"2d135c11-0f07-413e-b7d1-94956048825f","Type":"ContainerStarted","Data":"34eb8dc58379adf04e4475ebe695b247ef1f1f15cf6ed73b7ed0c163bcb46cad"} Nov 02 08:43:11.457966 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:11.457796 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" event={"ID":"2d135c11-0f07-413e-b7d1-94956048825f","Type":"ContainerStarted","Data":"5a97cd9d33269b9c1910cc39ad619d53896c0d8c5485a08a605bb337a8dc0efb"} Nov 02 08:43:11.564957 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:11.564898 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" podStartSLOduration=2.5648832820000003 podStartE2EDuration="2.564883282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:11.562973688 +0000 UTC m=+2238.653594107" watchObservedRunningTime="2026-11-02 08:43:11.564883282 +0000 UTC m=+2238.655503700" Nov 02 08:43:12.468567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:12.468524 5055 generic.go:356] "Generic (PLEG): container finished" podID="2d135c11-0f07-413e-b7d1-94956048825f" containerID="34eb8dc58379adf04e4475ebe695b247ef1f1f15cf6ed73b7ed0c163bcb46cad" exitCode=0 Nov 02 08:43:12.469082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:12.468603 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" event={"ID":"2d135c11-0f07-413e-b7d1-94956048825f","Type":"ContainerDied","Data":"34eb8dc58379adf04e4475ebe695b247ef1f1f15cf6ed73b7ed0c163bcb46cad"} Nov 02 08:43:13.651736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.651705 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:13.729217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.729091 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca\") pod \"2d135c11-0f07-413e-b7d1-94956048825f\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " Nov 02 08:43:13.729217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.729172 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca\") pod \"2d135c11-0f07-413e-b7d1-94956048825f\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " Nov 02 08:43:13.729464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.729368 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm\" (UniqueName: \"kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm\") pod \"2d135c11-0f07-413e-b7d1-94956048825f\" (UID: \"2d135c11-0f07-413e-b7d1-94956048825f\") " Nov 02 08:43:13.729573 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.729544 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca" pod "2d135c11-0f07-413e-b7d1-94956048825f" (UID: "2d135c11-0f07-413e-b7d1-94956048825f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:13.729876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.729851 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:13.731443 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.731420 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca" pod "2d135c11-0f07-413e-b7d1-94956048825f" (UID: "2d135c11-0f07-413e-b7d1-94956048825f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:13.731552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.731449 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm" pod "2d135c11-0f07-413e-b7d1-94956048825f" (UID: "2d135c11-0f07-413e-b7d1-94956048825f"). InnerVolumeSpecName "kube-api-access-fxztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:13.831131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.831088 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d135c11-0f07-413e-b7d1-94956048825f-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:13.831131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:13.831133 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxztm\" (UniqueName: \"kubernetes.io/projected/2d135c11-0f07-413e-b7d1-94956048825f-kube-api-access-fxztm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:14.485915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:14.485884 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" Nov 02 08:43:14.486132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:14.485887 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp" event={"ID":"2d135c11-0f07-413e-b7d1-94956048825f","Type":"ContainerDied","Data":"5a97cd9d33269b9c1910cc39ad619d53896c0d8c5485a08a605bb337a8dc0efb"} Nov 02 08:43:14.486132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:14.486017 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a97cd9d33269b9c1910cc39ad619d53896c0d8c5485a08a605bb337a8dc0efb" Nov 02 08:43:15.888089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:15.888049 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:43:21.384469 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:21.384432 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5"] Nov 02 08:43:24.922420 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:43:24.922381 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:43:26.385095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.385038 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" podUID="57559d64-6360-4b38-9ad7-0edde9d7ed18" containerName="payload-pre-processing" containerID="cri-o://2fe1fda5f04ad73616bf5d93f6c21432ebf2d9cc5afd1deab7d6bb98bdf697a8" gracePeriod=25 Nov 02 08:43:26.602702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.602455 5055 generic.go:356] "Generic (PLEG): container finished" podID="57559d64-6360-4b38-9ad7-0edde9d7ed18" containerID="2fe1fda5f04ad73616bf5d93f6c21432ebf2d9cc5afd1deab7d6bb98bdf697a8" exitCode=0 Nov 02 08:43:26.602702 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.602625 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" event={"ID":"57559d64-6360-4b38-9ad7-0edde9d7ed18","Type":"ContainerDied","Data":"2fe1fda5f04ad73616bf5d93f6c21432ebf2d9cc5afd1deab7d6bb98bdf697a8"} Nov 02 08:43:26.696674 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.696639 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:43:26.854548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.854504 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28\" (UniqueName: \"kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28\") pod \"57559d64-6360-4b38-9ad7-0edde9d7ed18\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " Nov 02 08:43:26.854749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.854587 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume\") pod \"57559d64-6360-4b38-9ad7-0edde9d7ed18\" (UID: \"57559d64-6360-4b38-9ad7-0edde9d7ed18\") " Nov 02 08:43:26.855287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.855232 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume" pod "57559d64-6360-4b38-9ad7-0edde9d7ed18" (UID: "57559d64-6360-4b38-9ad7-0edde9d7ed18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:26.860677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.860645 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28" pod "57559d64-6360-4b38-9ad7-0edde9d7ed18" (UID: "57559d64-6360-4b38-9ad7-0edde9d7ed18"). InnerVolumeSpecName "kube-api-access-smg28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:26.956730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.956693 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smg28\" (UniqueName: \"kubernetes.io/projected/57559d64-6360-4b38-9ad7-0edde9d7ed18-kube-api-access-smg28\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:26.956730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:26.956725 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57559d64-6360-4b38-9ad7-0edde9d7ed18-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:27.616548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:27.616516 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" Nov 02 08:43:27.616548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:27.616535 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5" event={"ID":"57559d64-6360-4b38-9ad7-0edde9d7ed18","Type":"ContainerDied","Data":"a1192b5a265994abccbc1e1aa3c388ad176d8823bdfbf1d07601c753dcadd742"} Nov 02 08:43:27.617086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:27.616607 5055 scope.go:122] "RemoveContainer" containerID="2fe1fda5f04ad73616bf5d93f6c21432ebf2d9cc5afd1deab7d6bb98bdf697a8" Nov 02 08:43:27.770320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:27.770283 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5"] Nov 02 08:43:27.829604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:27.829385 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4516e-55c767bf75-v5lj5"] Nov 02 08:43:29.136283 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:29.136242 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57559d64-6360-4b38-9ad7-0edde9d7ed18" path="/var/lib/kubelet/pods/57559d64-6360-4b38-9ad7-0edde9d7ed18/volumes" Nov 02 08:43:29.200722 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:29.200679 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp"] Nov 02 08:43:29.242863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:29.242831 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4516e-d5zfp"] Nov 02 08:43:31.095220 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:31.095134 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d135c11-0f07-413e-b7d1-94956048825f" path="/var/lib/kubelet/pods/2d135c11-0f07-413e-b7d1-94956048825f/volumes" Nov 02 08:43:36.626364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.626326 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99"] Nov 02 08:43:36.626905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.626889 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2d135c11-0f07-413e-b7d1-94956048825f" containerName="revoke-keys" Nov 02 08:43:36.626905 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.626902 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d135c11-0f07-413e-b7d1-94956048825f" containerName="revoke-keys" Nov 02 08:43:36.626983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.626942 5055 cpu_manager.go:414] "containerMap: removing container" podUID="57559d64-6360-4b38-9ad7-0edde9d7ed18" containerName="payload-pre-processing" Nov 02 08:43:36.626983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.626948 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57559d64-6360-4b38-9ad7-0edde9d7ed18" containerName="payload-pre-processing" Nov 02 08:43:36.627056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.627028 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d135c11-0f07-413e-b7d1-94956048825f" containerName="revoke-keys" Nov 02 08:43:36.627056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.627046 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="57559d64-6360-4b38-9ad7-0edde9d7ed18" containerName="payload-pre-processing" Nov 02 08:43:36.632211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.632190 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.684450 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.684414 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:43:36.713761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.713725 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7f9\" (UniqueName: \"kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.713950 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.713853 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.713995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.713981 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.730175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.730103 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:43:36.730358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.730217 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:43:36.730547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.730524 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:43:36.734095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.734075 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99"] Nov 02 08:43:36.815575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.815541 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.815769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.815651 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7f9\" (UniqueName: \"kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.815769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.815745 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.816350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.816323 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.818278 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.818255 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.869279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.869243 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7f9\" (UniqueName: \"kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:36.946575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:36.946479 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:37.320103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:37.320069 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99"] Nov 02 08:43:37.321229 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:43:37.321196 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87e6f45_099d_484f_a8b8_05bcd25190a0.slice/crio-c5a2e656b0b0117b5c37a71bd05e52c48a61fa838110572f7cb984b0686e3a2e WatchSource:0}: Error finding container c5a2e656b0b0117b5c37a71bd05e52c48a61fa838110572f7cb984b0686e3a2e: Status 404 returned error can't find the container with id c5a2e656b0b0117b5c37a71bd05e52c48a61fa838110572f7cb984b0686e3a2e Nov 02 08:43:37.698987 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:37.698946 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" event={"ID":"c87e6f45-099d-484f-a8b8-05bcd25190a0","Type":"ContainerStarted","Data":"b86f94db2cb48b2ccca778583976e2111b9f7e14c32e893ce2d8b5a081487c59"} Nov 02 08:43:37.699672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:37.699007 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" event={"ID":"c87e6f45-099d-484f-a8b8-05bcd25190a0","Type":"ContainerStarted","Data":"c5a2e656b0b0117b5c37a71bd05e52c48a61fa838110572f7cb984b0686e3a2e"} Nov 02 08:43:37.806445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:37.806390 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" podStartSLOduration=1.80637412 podStartE2EDuration="1.80637412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:37.804652108 +0000 UTC m=+2264.895272600" watchObservedRunningTime="2026-11-02 08:43:37.80637412 +0000 UTC m=+2264.896994570" Nov 02 08:43:38.709200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:38.709129 5055 generic.go:356] "Generic (PLEG): container finished" podID="c87e6f45-099d-484f-a8b8-05bcd25190a0" containerID="b86f94db2cb48b2ccca778583976e2111b9f7e14c32e893ce2d8b5a081487c59" exitCode=0 Nov 02 08:43:38.709641 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:38.709207 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" event={"ID":"c87e6f45-099d-484f-a8b8-05bcd25190a0","Type":"ContainerDied","Data":"b86f94db2cb48b2ccca778583976e2111b9f7e14c32e893ce2d8b5a081487c59"} Nov 02 08:43:39.882250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.882219 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:39.954857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.954790 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca\") pod \"c87e6f45-099d-484f-a8b8-05bcd25190a0\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " Nov 02 08:43:39.955070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.954949 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca\") pod \"c87e6f45-099d-484f-a8b8-05bcd25190a0\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " Nov 02 08:43:39.955282 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.955257 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9\" (UniqueName: \"kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9\") pod \"c87e6f45-099d-484f-a8b8-05bcd25190a0\" (UID: \"c87e6f45-099d-484f-a8b8-05bcd25190a0\") " Nov 02 08:43:39.955473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.955422 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca" pod "c87e6f45-099d-484f-a8b8-05bcd25190a0" (UID: "c87e6f45-099d-484f-a8b8-05bcd25190a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:39.955884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.955861 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:39.957518 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.957492 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9" pod "c87e6f45-099d-484f-a8b8-05bcd25190a0" (UID: "c87e6f45-099d-484f-a8b8-05bcd25190a0"). InnerVolumeSpecName "kube-api-access-5p7f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:39.957619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:39.957589 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca" pod "c87e6f45-099d-484f-a8b8-05bcd25190a0" (UID: "c87e6f45-099d-484f-a8b8-05bcd25190a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:40.057239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:40.057099 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87e6f45-099d-484f-a8b8-05bcd25190a0-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:40.057239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:40.057183 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7f9\" (UniqueName: \"kubernetes.io/projected/c87e6f45-099d-484f-a8b8-05bcd25190a0-kube-api-access-5p7f9\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:40.739346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:40.739312 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" Nov 02 08:43:40.739554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:40.739338 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99" event={"ID":"c87e6f45-099d-484f-a8b8-05bcd25190a0","Type":"ContainerDied","Data":"c5a2e656b0b0117b5c37a71bd05e52c48a61fa838110572f7cb984b0686e3a2e"} Nov 02 08:43:40.739554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:40.739411 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5a2e656b0b0117b5c37a71bd05e52c48a61fa838110572f7cb984b0686e3a2e" Nov 02 08:43:42.687042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.687001 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb"] Nov 02 08:43:42.687589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.687572 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c87e6f45-099d-484f-a8b8-05bcd25190a0" containerName="revoke-keys" Nov 02 08:43:42.687651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.687590 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87e6f45-099d-484f-a8b8-05bcd25190a0" containerName="revoke-keys" Nov 02 08:43:42.687726 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.687715 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87e6f45-099d-484f-a8b8-05bcd25190a0" containerName="revoke-keys" Nov 02 08:43:42.691074 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.691050 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.739752 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.739722 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:43:42.739969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.739946 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:43:42.785880 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.785843 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:43:42.786092 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.785910 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:43:42.789745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.789719 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb"] Nov 02 08:43:42.792633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.792609 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.792765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.792669 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.792969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.792941 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8n7q\" (UniqueName: \"kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.894152 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.894091 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.894370 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.894184 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.894436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.894364 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8n7q\" (UniqueName: \"kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.894835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.894809 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.896769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.896734 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:42.947496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:42.947424 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8n7q\" (UniqueName: \"kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q\") pod \"maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:43.006623 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:43.006586 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:43.378603 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:43.378573 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb"] Nov 02 08:43:43.379926 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:43:43.379894 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679b6d8a_8fd8_4764_be2d_edd33d55a2f1.slice/crio-6fa8d3fd8f0e08a1a3f1f1378357667965615ca5e2380bc6bd5c5c82c834d485 WatchSource:0}: Error finding container 6fa8d3fd8f0e08a1a3f1f1378357667965615ca5e2380bc6bd5c5c82c834d485: Status 404 returned error can't find the container with id 6fa8d3fd8f0e08a1a3f1f1378357667965615ca5e2380bc6bd5c5c82c834d485 Nov 02 08:43:43.765875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:43.765837 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" event={"ID":"679b6d8a-8fd8-4764-be2d-edd33d55a2f1","Type":"ContainerStarted","Data":"0fb6c7af5987845b6df06c69708282532f8c6fd8e0066c7942d0150bb0d9ffa4"} Nov 02 08:43:43.766329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:43.765905 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" event={"ID":"679b6d8a-8fd8-4764-be2d-edd33d55a2f1","Type":"ContainerStarted","Data":"6fa8d3fd8f0e08a1a3f1f1378357667965615ca5e2380bc6bd5c5c82c834d485"} Nov 02 08:43:43.872174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:43.872089 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" podStartSLOduration=1.8720755150000001 podStartE2EDuration="1.872075515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:43.871339356 +0000 UTC m=+2270.961959774" watchObservedRunningTime="2026-11-02 08:43:43.872075515 +0000 UTC m=+2270.962695933" Nov 02 08:43:44.775337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:44.775300 5055 generic.go:356] "Generic (PLEG): container finished" podID="679b6d8a-8fd8-4764-be2d-edd33d55a2f1" containerID="0fb6c7af5987845b6df06c69708282532f8c6fd8e0066c7942d0150bb0d9ffa4" exitCode=0 Nov 02 08:43:44.775779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:44.775365 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" event={"ID":"679b6d8a-8fd8-4764-be2d-edd33d55a2f1","Type":"ContainerDied","Data":"0fb6c7af5987845b6df06c69708282532f8c6fd8e0066c7942d0150bb0d9ffa4"} Nov 02 08:43:46.033723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.033684 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:46.138160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.137552 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q\" (UniqueName: \"kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q\") pod \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " Nov 02 08:43:46.138160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.137714 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca\") pod \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " Nov 02 08:43:46.138160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.137905 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca\") pod \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\" (UID: \"679b6d8a-8fd8-4764-be2d-edd33d55a2f1\") " Nov 02 08:43:46.154199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.140400 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca" pod "679b6d8a-8fd8-4764-be2d-edd33d55a2f1" (UID: "679b6d8a-8fd8-4764-be2d-edd33d55a2f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:46.154199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.144644 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca" pod "679b6d8a-8fd8-4764-be2d-edd33d55a2f1" (UID: "679b6d8a-8fd8-4764-be2d-edd33d55a2f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:46.154199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.147133 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q" pod "679b6d8a-8fd8-4764-be2d-edd33d55a2f1" (UID: "679b6d8a-8fd8-4764-be2d-edd33d55a2f1"). InnerVolumeSpecName "kube-api-access-n8n7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:46.197562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.197530 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:43:46.240123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.240020 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:46.240123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.240055 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8n7q\" (UniqueName: \"kubernetes.io/projected/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-kube-api-access-n8n7q\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:46.240123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.240074 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679b6d8a-8fd8-4764-be2d-edd33d55a2f1-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:46.792303 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.792266 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" Nov 02 08:43:46.792497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.792304 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb" event={"ID":"679b6d8a-8fd8-4764-be2d-edd33d55a2f1","Type":"ContainerDied","Data":"6fa8d3fd8f0e08a1a3f1f1378357667965615ca5e2380bc6bd5c5c82c834d485"} Nov 02 08:43:46.792497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:46.792367 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa8d3fd8f0e08a1a3f1f1378357667965615ca5e2380bc6bd5c5c82c834d485" Nov 02 08:43:47.206036 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:47.206000 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq"] Nov 02 08:43:47.252740 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:47.252707 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl"] Nov 02 08:43:47.931263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:47.931207 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq"] Nov 02 08:43:47.934846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:47.934815 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e6407-hnzhq"] Nov 02 08:43:49.092368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:49.092332 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54392f8b-b6a0-4e47-ad9c-a85499485a63" path="/var/lib/kubelet/pods/54392f8b-b6a0-4e47-ad9c-a85499485a63/volumes" Nov 02 08:43:54.924738 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:43:54.924693 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:43:55.692795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:55.692759 5055 scope.go:122] "RemoveContainer" containerID="5033f4c6b928793cd4a397d8f19432b1d4d7788cfc0763bee2c7ca2252c0031d" Nov 02 08:43:55.706475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:43:55.706327 5055 scope.go:122] "RemoveContainer" containerID="4fc4c904ec4bb50c7b79e54a6048ad742c9f1b0386d618dfdb8683ec651fc758" Nov 02 08:44:02.228455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.228410 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" containerID="cri-o://5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912" gracePeriod=45 Nov 02 08:44:02.274800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.274761 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" containerID="cri-o://0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3" gracePeriod=45 Nov 02 08:44:02.560601 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.560573 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:44:02.591401 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.591373 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:44:02.691614 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691570 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir\") pod \"c30389c6-bcc7-4fd4-8498-11094b658732\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " Nov 02 08:44:02.691816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691643 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home\") pod \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " Nov 02 08:44:02.691816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691773 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n\" (UniqueName: \"kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n\") pod \"c30389c6-bcc7-4fd4-8498-11094b658732\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " Nov 02 08:44:02.691947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691843 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs\") pod \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " Nov 02 08:44:02.691947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691906 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home" pod "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" (UID: "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.691947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691912 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir" pod "c30389c6-bcc7-4fd4-8498-11094b658732" (UID: "c30389c6-bcc7-4fd4-8498-11094b658732"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.692071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691971 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home\") pod \"c30389c6-bcc7-4fd4-8498-11094b658732\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " Nov 02 08:44:02.692071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.691999 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm\") pod \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " Nov 02 08:44:02.692246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692182 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache\") pod \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " Nov 02 08:44:02.692246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692220 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs\" (UniqueName: \"kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs\") pod \"c30389c6-bcc7-4fd4-8498-11094b658732\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " Nov 02 08:44:02.692363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692256 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home" pod "c30389c6-bcc7-4fd4-8498-11094b658732" (UID: "c30389c6-bcc7-4fd4-8498-11094b658732"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.692363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692290 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps\" (UniqueName: \"kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps\") pod \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " Nov 02 08:44:02.692483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692364 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir\") pod \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\" (UID: \"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f\") " Nov 02 08:44:02.692483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692390 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache\") pod \"c30389c6-bcc7-4fd4-8498-11094b658732\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " Nov 02 08:44:02.692483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692416 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm\") pod \"c30389c6-bcc7-4fd4-8498-11094b658732\" (UID: \"c30389c6-bcc7-4fd4-8498-11094b658732\") " Nov 02 08:44:02.692483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692472 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache" pod "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" (UID: "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.692695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692631 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache" pod "c30389c6-bcc7-4fd4-8498-11094b658732" (UID: "c30389c6-bcc7-4fd4-8498-11094b658732"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.693026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.692999 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.693119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.693030 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.693119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.693048 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.693119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.693062 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.693119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.693074 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.693374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.693129 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir" pod "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" (UID: "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.694674 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.694648 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n" pod "c30389c6-bcc7-4fd4-8498-11094b658732" (UID: "c30389c6-bcc7-4fd4-8498-11094b658732"). InnerVolumeSpecName "kube-api-access-wmp5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:02.694839 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.694815 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs" pod "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" (UID: "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:02.694976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.694950 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs" pod "c30389c6-bcc7-4fd4-8498-11094b658732" (UID: "c30389c6-bcc7-4fd4-8498-11094b658732"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:02.695201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.695109 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm" pod "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" (UID: "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.695296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.695268 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps" pod "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" (UID: "d33b3ab9-b4bc-4f43-af95-313f6e8cda4f"). InnerVolumeSpecName "kube-api-access-x6nps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:02.695345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.695330 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm" pod "c30389c6-bcc7-4fd4-8498-11094b658732" (UID: "c30389c6-bcc7-4fd4-8498-11094b658732"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:02.798150 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798048 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.799049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798447 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30389c6-bcc7-4fd4-8498-11094b658732-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.799049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798478 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6nps\" (UniqueName: \"kubernetes.io/projected/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-kube-api-access-x6nps\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.799049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798543 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.799049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798570 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30389c6-bcc7-4fd4-8498-11094b658732-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.799049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798585 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmp5n\" (UniqueName: \"kubernetes.io/projected/c30389c6-bcc7-4fd4-8498-11094b658732-kube-api-access-wmp5n\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:02.799049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:02.798599 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:03.046535 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.046496 5055 generic.go:356] "Generic (PLEG): container finished" podID="c30389c6-bcc7-4fd4-8498-11094b658732" containerID="5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912" exitCode=0 Nov 02 08:44:03.046743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.046570 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" Nov 02 08:44:03.046743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.046585 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" event={"ID":"c30389c6-bcc7-4fd4-8498-11094b658732","Type":"ContainerDied","Data":"5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912"} Nov 02 08:44:03.046743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.046634 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" event={"ID":"c30389c6-bcc7-4fd4-8498-11094b658732","Type":"ContainerDied","Data":"1b876124ae9c0ef1667cc13faf8a8388d0dc0d3d63bde2c92c7ac7e88a809ba4"} Nov 02 08:44:03.046743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.046658 5055 scope.go:122] "RemoveContainer" containerID="5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912" Nov 02 08:44:03.048478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.048406 5055 generic.go:356] "Generic (PLEG): container finished" podID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerID="0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3" exitCode=0 Nov 02 08:44:03.048478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.048454 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" Nov 02 08:44:03.048662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.048455 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" event={"ID":"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f","Type":"ContainerDied","Data":"0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3"} Nov 02 08:44:03.048662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.048568 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl" event={"ID":"d33b3ab9-b4bc-4f43-af95-313f6e8cda4f","Type":"ContainerDied","Data":"54944e946cf4eee396f150a8b9811f55182a82d8cf77f1fa7de664eb85bd2080"} Nov 02 08:44:03.057909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.057886 5055 scope.go:122] "RemoveContainer" containerID="5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912" Nov 02 08:44:03.058296 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:03.058269 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912\": container with ID starting with 5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912 not found: ID does not exist" containerID="5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912" Nov 02 08:44:03.058420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.058310 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912"} err="failed to get container status \"5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912\": rpc error: code = NotFound desc = could not find container \"5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912\": container with ID starting with 5571df12fb431185800497c94cc4b5c48fdd6147318cc5b03d76424b1a619912 not found: ID does not exist" Nov 02 08:44:03.058420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.058336 5055 scope.go:122] "RemoveContainer" containerID="0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3" Nov 02 08:44:03.068340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.068318 5055 scope.go:122] "RemoveContainer" containerID="0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3" Nov 02 08:44:03.068631 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:03.068609 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3\": container with ID starting with 0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3 not found: ID does not exist" containerID="0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3" Nov 02 08:44:03.068683 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.068640 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3"} err="failed to get container status \"0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3\": rpc error: code = NotFound desc = could not find container \"0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3\": container with ID starting with 0249bd75007baf26a23339c1d0dfc6e6348b93bb292c8cf60ac0b959c1d08ec3 not found: ID does not exist" Nov 02 08:44:03.204737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.204705 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq"] Nov 02 08:44:03.209291 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.209266 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq"] Nov 02 08:44:03.323607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.323509 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl"] Nov 02 08:44:03.328644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.328609 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e6ca80/distinct-w0-e6ca80-kserve-769b7745c4-dswfl"] Nov 02 08:44:03.448677 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:03.448631 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e6ca80/distinct-2-w0-e6ca80-kserve-6cf9d567bf-dggsq" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:44:04.454535 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:04.454445 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5"] Nov 02 08:44:04.454963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:04.454702 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" containerID="cri-o://7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794" gracePeriod=30 Nov 02 08:44:05.095800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:05.095753 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" path="/var/lib/kubelet/pods/c30389c6-bcc7-4fd4-8498-11094b658732/volumes" Nov 02 08:44:05.096484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:05.096458 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" path="/var/lib/kubelet/pods/d33b3ab9-b4bc-4f43-af95-313f6e8cda4f/volumes" Nov 02 08:44:09.735862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.735795 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:44:09.799557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799505 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799569 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799604 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799644 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799804 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799857 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799898 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.799943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799933 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:09.800185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.799940 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.800185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800021 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy\") pod \"3b2211da-e856-427e-946c-6c7b14714377\" (UID: \"3b2211da-e856-427e-946c-6c7b14714377\") " Nov 02 08:44:09.800185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800160 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:09.800346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800211 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:09.800404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800354 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:09.800495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800451 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:09.800746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800717 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.800746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800741 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.800912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800756 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.800912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800771 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.800912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.800785 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b2211da-e856-427e-946c-6c7b14714377-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.802698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.802664 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:44:09.802823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.802755 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "kube-api-access-cj9h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:09.802975 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.802944 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:09.803074 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.803040 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token" pod "3b2211da-e856-427e-946c-6c7b14714377" (UID: "3b2211da-e856-427e-946c-6c7b14714377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:09.902094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.902023 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.902094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.902087 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj9h4\" (UniqueName: \"kubernetes.io/projected/3b2211da-e856-427e-946c-6c7b14714377-kube-api-access-cj9h4\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.902415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.902106 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b2211da-e856-427e-946c-6c7b14714377-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:09.902415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:09.902120 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b2211da-e856-427e-946c-6c7b14714377-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:10.106357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.106259 5055 generic.go:356] "Generic (PLEG): container finished" podID="3b2211da-e856-427e-946c-6c7b14714377" containerID="7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794" exitCode=0 Nov 02 08:44:10.106357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.106328 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" Nov 02 08:44:10.106357 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.106344 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" event={"ID":"3b2211da-e856-427e-946c-6c7b14714377","Type":"ContainerDied","Data":"7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794"} Nov 02 08:44:10.106662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.106405 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5" event={"ID":"3b2211da-e856-427e-946c-6c7b14714377","Type":"ContainerDied","Data":"7cb68a4273b2cec99dfa8b787091346bf655b27ad6dafc2aabfc2c4097c3413e"} Nov 02 08:44:10.106662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.106429 5055 scope.go:122] "RemoveContainer" containerID="7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794" Nov 02 08:44:10.118560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.118532 5055 scope.go:122] "RemoveContainer" containerID="285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8" Nov 02 08:44:10.131522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.131487 5055 scope.go:122] "RemoveContainer" containerID="7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794" Nov 02 08:44:10.131914 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:10.131892 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794\": container with ID starting with 7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794 not found: ID does not exist" containerID="7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794" Nov 02 08:44:10.131961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.131927 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794"} err="failed to get container status \"7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794\": rpc error: code = NotFound desc = could not find container \"7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794\": container with ID starting with 7c85cfc765e36764090af1d9cc162f2a0cd7853c8c8ccee92292f1e14bb6c794 not found: ID does not exist" Nov 02 08:44:10.131961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.131948 5055 scope.go:122] "RemoveContainer" containerID="285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8" Nov 02 08:44:10.132270 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:10.132246 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8\": container with ID starting with 285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8 not found: ID does not exist" containerID="285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8" Nov 02 08:44:10.132348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.132278 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8"} err="failed to get container status \"285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8\": rpc error: code = NotFound desc = could not find container \"285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8\": container with ID starting with 285d1ebd00a9101967c71bab7f499069d83273c19edd56a4a07205f12313b0f8 not found: ID does not exist" Nov 02 08:44:10.261211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.261166 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5"] Nov 02 08:44:10.265308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:10.265276 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-7lrl5"] Nov 02 08:44:11.094552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:11.094511 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2211da-e856-427e-946c-6c7b14714377" path="/var/lib/kubelet/pods/3b2211da-e856-427e-946c-6c7b14714377/volumes" Nov 02 08:44:14.112679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:14.112636 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:44:21.295980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.295943 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d"] Nov 02 08:44:21.296649 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296629 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" Nov 02 08:44:21.296727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296651 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" Nov 02 08:44:21.296727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296682 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" Nov 02 08:44:21.296727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296689 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" Nov 02 08:44:21.296727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296712 5055 cpu_manager.go:414] "containerMap: removing container" podUID="679b6d8a-8fd8-4764-be2d-edd33d55a2f1" containerName="revoke-keys" Nov 02 08:44:21.296727 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296718 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679b6d8a-8fd8-4764-be2d-edd33d55a2f1" containerName="revoke-keys" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296729 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296741 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296760 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296767 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296915 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30389c6-bcc7-4fd4-8498-11094b658732" containerName="main" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296934 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33b3ab9-b4bc-4f43-af95-313f6e8cda4f" containerName="main" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296957 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296969 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="679b6d8a-8fd8-4764-be2d-edd33d55a2f1" containerName="revoke-keys" Nov 02 08:44:21.296999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.296977 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2211da-e856-427e-946c-6c7b14714377" containerName="istio-proxy" Nov 02 08:44:21.301447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.301423 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.332585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.332548 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.332769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.332616 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.332769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.332745 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.351643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.351613 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh"] Nov 02 08:44:21.355725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.355696 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.394543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.394467 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:44:21.394543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.394523 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:44:21.396085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.396010 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:44:21.397046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.396105 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:44:21.399885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.399860 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d"] Nov 02 08:44:21.434111 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434081 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.434316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434258 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.434316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434285 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.434495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434343 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.434562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434519 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.434633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434572 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttz6r\" (UniqueName: \"kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.434996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.434964 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.436989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.436958 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.453729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.453683 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh"] Nov 02 08:44:21.487951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.487909 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7\") pod \"maas-api-revoke-sub-2d46e07b-jdb7d\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.535902 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.535864 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.536119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.535910 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttz6r\" (UniqueName: \"kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.536119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.536006 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.536788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.536762 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.538582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.538552 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.589562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.589485 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttz6r\" (UniqueName: \"kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r\") pod \"maas-api-revoke-sub-55df2580-ndgsh\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:21.620787 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.620751 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:21.690194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:21.671414 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:22.037652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:22.037611 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d"] Nov 02 08:44:22.039933 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:44:22.039902 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a8f263_2f7f_4c60_b5e8_1e321cb4a30f.slice/crio-4c1ebfaa1a931b0cd66afcdc4d4b89f2cea2ddb0cc7a12c8cb07f2be6074dc5f WatchSource:0}: Error finding container 4c1ebfaa1a931b0cd66afcdc4d4b89f2cea2ddb0cc7a12c8cb07f2be6074dc5f: Status 404 returned error can't find the container with id 4c1ebfaa1a931b0cd66afcdc4d4b89f2cea2ddb0cc7a12c8cb07f2be6074dc5f Nov 02 08:44:22.132241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:22.132190 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh"] Nov 02 08:44:22.133976 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:44:22.133938 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed55156_814c_4560_bffa_e2aa9168e440.slice/crio-3e9513a2d54fd94ad52666c597212151cb079556b63230b0f3094a7eb19cd203 WatchSource:0}: Error finding container 3e9513a2d54fd94ad52666c597212151cb079556b63230b0f3094a7eb19cd203: Status 404 returned error can't find the container with id 3e9513a2d54fd94ad52666c597212151cb079556b63230b0f3094a7eb19cd203 Nov 02 08:44:22.237473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:22.237435 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" event={"ID":"aed55156-814c-4560-bffa-e2aa9168e440","Type":"ContainerStarted","Data":"3e9513a2d54fd94ad52666c597212151cb079556b63230b0f3094a7eb19cd203"} Nov 02 08:44:22.238795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:22.238750 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" event={"ID":"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f","Type":"ContainerStarted","Data":"4c1ebfaa1a931b0cd66afcdc4d4b89f2cea2ddb0cc7a12c8cb07f2be6074dc5f"} Nov 02 08:44:23.248877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:23.248845 5055 generic.go:356] "Generic (PLEG): container finished" podID="aed55156-814c-4560-bffa-e2aa9168e440" containerID="d3878ca1d191ec99b6651b5d20ccbf81119ef9f66ad837a44945756842be81ef" exitCode=0 Nov 02 08:44:23.249312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:23.248935 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" event={"ID":"aed55156-814c-4560-bffa-e2aa9168e440","Type":"ContainerDied","Data":"d3878ca1d191ec99b6651b5d20ccbf81119ef9f66ad837a44945756842be81ef"} Nov 02 08:44:23.251901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:23.251871 5055 generic.go:356] "Generic (PLEG): container finished" podID="a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" containerID="e80350a670b23fa6a1a2412e9a479c073893df12863d2a78c0090612138a8fec" exitCode=0 Nov 02 08:44:23.252029 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:23.251957 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" event={"ID":"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f","Type":"ContainerDied","Data":"e80350a670b23fa6a1a2412e9a479c073893df12863d2a78c0090612138a8fec"} Nov 02 08:44:24.527565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.527530 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:24.530998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.530970 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:24.618652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.618614 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca\") pod \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " Nov 02 08:44:24.618863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.618831 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7\") pod \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " Nov 02 08:44:24.618936 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.618894 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca\") pod \"aed55156-814c-4560-bffa-e2aa9168e440\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " Nov 02 08:44:24.619000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.618957 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r\" (UniqueName: \"kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r\") pod \"aed55156-814c-4560-bffa-e2aa9168e440\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " Nov 02 08:44:24.619057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.619033 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca\") pod \"aed55156-814c-4560-bffa-e2aa9168e440\" (UID: \"aed55156-814c-4560-bffa-e2aa9168e440\") " Nov 02 08:44:24.619112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.619084 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca\") pod \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\" (UID: \"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f\") " Nov 02 08:44:24.619580 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.619551 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca" pod "aed55156-814c-4560-bffa-e2aa9168e440" (UID: "aed55156-814c-4560-bffa-e2aa9168e440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:24.619684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.619607 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca" pod "a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" (UID: "a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:24.621322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.621295 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r" pod "aed55156-814c-4560-bffa-e2aa9168e440" (UID: "aed55156-814c-4560-bffa-e2aa9168e440"). InnerVolumeSpecName "kube-api-access-ttz6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:24.621419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.621354 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca" pod "a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" (UID: "a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:24.621560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.621541 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7" pod "a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" (UID: "a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f"). InnerVolumeSpecName "kube-api-access-wh4j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:24.621618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.621593 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca" pod "aed55156-814c-4560-bffa-e2aa9168e440" (UID: "aed55156-814c-4560-bffa-e2aa9168e440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:24.720767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.720714 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:24.720767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.720744 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-kube-api-access-wh4j7\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:24.720767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.720755 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:24.720767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.720764 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttz6r\" (UniqueName: \"kubernetes.io/projected/aed55156-814c-4560-bffa-e2aa9168e440-kube-api-access-ttz6r\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:24.720767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.720774 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed55156-814c-4560-bffa-e2aa9168e440-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:24.721115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:24.720783 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:24.922626 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:24.922540 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:44:25.269200 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.269160 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" event={"ID":"aed55156-814c-4560-bffa-e2aa9168e440","Type":"ContainerDied","Data":"3e9513a2d54fd94ad52666c597212151cb079556b63230b0f3094a7eb19cd203"} Nov 02 08:44:25.269485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.269228 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9513a2d54fd94ad52666c597212151cb079556b63230b0f3094a7eb19cd203" Nov 02 08:44:25.269485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.269178 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh" Nov 02 08:44:25.270921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.270891 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" Nov 02 08:44:25.271060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.270914 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d" event={"ID":"a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f","Type":"ContainerDied","Data":"4c1ebfaa1a931b0cd66afcdc4d4b89f2cea2ddb0cc7a12c8cb07f2be6074dc5f"} Nov 02 08:44:25.271060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.270962 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c1ebfaa1a931b0cd66afcdc4d4b89f2cea2ddb0cc7a12c8cb07f2be6074dc5f" Nov 02 08:44:25.285463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.285422 5055 generic.go:356] "Generic (PLEG): container finished" podID="058b922b-5389-4fc1-a032-1a49596e1f31" containerID="4d3a1fcc96bf01cb9e71941616f9f13cc285c1d6d2695b9e13085dabad67a3b9" exitCode=0 Nov 02 08:44:25.285666 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.285640 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" event={"ID":"058b922b-5389-4fc1-a032-1a49596e1f31","Type":"ContainerDied","Data":"4d3a1fcc96bf01cb9e71941616f9f13cc285c1d6d2695b9e13085dabad67a3b9"} Nov 02 08:44:25.286327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.286298 5055 scope.go:122] "RemoveContainer" containerID="4d3a1fcc96bf01cb9e71941616f9f13cc285c1d6d2695b9e13085dabad67a3b9" Nov 02 08:44:25.288763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.288716 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:44:25.288909 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.288887 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:44:25.289010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:25.288993 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:44:26.316440 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:26.316407 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" event={"ID":"058b922b-5389-4fc1-a032-1a49596e1f31","Type":"ContainerStarted","Data":"9ce4ac1a01829d966d79646f4298bae11e1248ce1d566fe95246a24c40d61374"} Nov 02 08:44:27.276302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:27.276263 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:44:27.276684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:27.276662 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 08:44:27.276751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:27.276711 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 08:44:28.276245 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:28.276208 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 08:44:28.276788 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:28.276273 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 08:44:29.276207 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:29.276171 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 08:44:29.276415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:29.276227 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 08:44:30.276382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:30.276345 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 08:44:30.276585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:30.276400 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 08:44:31.276444 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:31.276411 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 08:44:31.276644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:31.276465 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 08:44:32.280589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:32.280552 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:44:32.281027 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:32.280946 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:44:32.281675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:32.281653 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:44:35.034403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:35.034359 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99"] Nov 02 08:44:35.079946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:35.079910 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad4580-n5m99"] Nov 02 08:44:35.094042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:35.093994 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87e6f45-099d-484f-a8b8-05bcd25190a0" path="/var/lib/kubelet/pods/c87e6f45-099d-484f-a8b8-05bcd25190a0/volumes" Nov 02 08:44:36.306294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:36.306254 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:44:41.615328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:41.615293 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd"] Nov 02 08:44:41.615785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:41.615497 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" containerID="cri-o://581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0" gracePeriod=30 Nov 02 08:44:42.596767 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.596729 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p"] Nov 02 08:44:42.597313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.597295 5055 cpu_manager.go:414] "containerMap: removing container" podUID="aed55156-814c-4560-bffa-e2aa9168e440" containerName="revoke-keys" Nov 02 08:44:42.597359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.597316 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed55156-814c-4560-bffa-e2aa9168e440" containerName="revoke-keys" Nov 02 08:44:42.597419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.597368 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" containerName="revoke-keys" Nov 02 08:44:42.597419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.597375 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" containerName="revoke-keys" Nov 02 08:44:42.597500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.597463 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed55156-814c-4560-bffa-e2aa9168e440" containerName="revoke-keys" Nov 02 08:44:42.597500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.597482 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" containerName="revoke-keys" Nov 02 08:44:42.602500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.602475 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.651873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.651836 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:44:42.652309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.651924 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:44:42.697847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.697815 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:44:42.698053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.697922 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:44:42.700808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.700776 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.700963 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.700888 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946rp\" (UniqueName: \"kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.701042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.701013 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.701672 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.701649 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p"] Nov 02 08:44:42.802213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.802171 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.802404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.802278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.802404 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.802312 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946rp\" (UniqueName: \"kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.803025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.802999 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.804644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.804620 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.856461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.856385 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946rp\" (UniqueName: \"kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp\") pod \"maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:42.917186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:42.917124 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:43.295980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:43.295938 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p"] Nov 02 08:44:43.300177 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:44:43.300129 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8c5900_0fbf_4735_a85c_6c91cc3aa95d.slice/crio-bdcd71ac770680394eee94277f0ab8c0ceadf62211b64181a06b6879f67cc48d WatchSource:0}: Error finding container bdcd71ac770680394eee94277f0ab8c0ceadf62211b64181a06b6879f67cc48d: Status 404 returned error can't find the container with id bdcd71ac770680394eee94277f0ab8c0ceadf62211b64181a06b6879f67cc48d Nov 02 08:44:43.455500 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:43.455448 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" event={"ID":"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d","Type":"ContainerStarted","Data":"bdcd71ac770680394eee94277f0ab8c0ceadf62211b64181a06b6879f67cc48d"} Nov 02 08:44:44.464821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:44.464774 5055 generic.go:356] "Generic (PLEG): container finished" podID="7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" containerID="0b7ac96698fdea4738b2056751ce7476e76c443c7cbe2882b1e287ea016dfa3c" exitCode=0 Nov 02 08:44:44.465511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:44.464859 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" event={"ID":"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d","Type":"ContainerDied","Data":"0b7ac96698fdea4738b2056751ce7476e76c443c7cbe2882b1e287ea016dfa3c"} Nov 02 08:44:45.635306 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.635261 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:45.747287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.747190 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp\" (UniqueName: \"kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp\") pod \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " Nov 02 08:44:45.747287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.747241 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca\") pod \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " Nov 02 08:44:45.747542 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.747383 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca\") pod \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\" (UID: \"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d\") " Nov 02 08:44:45.747799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.747733 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca" pod "7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" (UID: "7d8c5900-0fbf-4735-a85c-6c91cc3aa95d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:45.748052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.748027 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:45.749807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.749776 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca" pod "7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" (UID: "7d8c5900-0fbf-4735-a85c-6c91cc3aa95d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:45.750302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.750275 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp" pod "7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" (UID: "7d8c5900-0fbf-4735-a85c-6c91cc3aa95d"). InnerVolumeSpecName "kube-api-access-946rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:45.848841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.848784 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946rp\" (UniqueName: \"kubernetes.io/projected/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-kube-api-access-946rp\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:45.848841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:45.848836 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:46.484887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.484854 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" Nov 02 08:44:46.485086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.484888 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p" event={"ID":"7d8c5900-0fbf-4735-a85c-6c91cc3aa95d","Type":"ContainerDied","Data":"bdcd71ac770680394eee94277f0ab8c0ceadf62211b64181a06b6879f67cc48d"} Nov 02 08:44:46.485086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.484941 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdcd71ac770680394eee94277f0ab8c0ceadf62211b64181a06b6879f67cc48d" Nov 02 08:44:46.904749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.904725 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:44:46.960352 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960315 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960393 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960427 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960452 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960507 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960572 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960720 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960782 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960827 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket\") pod \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\" (UID: \"695546c1-9771-4a9b-a2bd-23ef70aa73f5\") " Nov 02 08:44:46.960996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.960715 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:46.961250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.961226 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:46.961398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.961379 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:46.961483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.961403 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:46.961483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.961464 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:46.961830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.961622 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:46.961830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.961782 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:46.964317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.964270 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:46.964317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.964287 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:44:46.964460 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.964309 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "kube-api-access-82lfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:46.964460 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:46.964343 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token" pod "695546c1-9771-4a9b-a2bd-23ef70aa73f5" (UID: "695546c1-9771-4a9b-a2bd-23ef70aa73f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:47.044682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.044592 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt"] Nov 02 08:44:47.063845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063809 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.063845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063846 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.064128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063863 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.064128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063880 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.064128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063895 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.064128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063911 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82lfx\" (UniqueName: \"kubernetes.io/projected/695546c1-9771-4a9b-a2bd-23ef70aa73f5-kube-api-access-82lfx\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.064128 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.063925 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695546c1-9771-4a9b-a2bd-23ef70aa73f5-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:47.068296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.068264 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb"] Nov 02 08:44:47.495474 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.495434 5055 generic.go:356] "Generic (PLEG): container finished" podID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerID="581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0" exitCode=0 Nov 02 08:44:47.495682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.495506 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" event={"ID":"695546c1-9771-4a9b-a2bd-23ef70aa73f5","Type":"ContainerDied","Data":"581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0"} Nov 02 08:44:47.495682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.495525 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" Nov 02 08:44:47.495682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.495598 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd" event={"ID":"695546c1-9771-4a9b-a2bd-23ef70aa73f5","Type":"ContainerDied","Data":"efe31fa6e96f1db3c328a0fc9cfaa759098c95b7256092f326029f5e551ca081"} Nov 02 08:44:47.495682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.495638 5055 scope.go:122] "RemoveContainer" containerID="581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0" Nov 02 08:44:47.506684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.506657 5055 scope.go:122] "RemoveContainer" containerID="d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c" Nov 02 08:44:47.517367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.517338 5055 scope.go:122] "RemoveContainer" containerID="581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0" Nov 02 08:44:47.517700 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:47.517673 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0\": container with ID starting with 581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0 not found: ID does not exist" containerID="581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0" Nov 02 08:44:47.517775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.517716 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0"} err="failed to get container status \"581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0\": rpc error: code = NotFound desc = could not find container \"581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0\": container with ID starting with 581d41a01b22ed08dbc2ec67798fad7226f56c435c5fa1f4ecf590f910d2c4e0 not found: ID does not exist" Nov 02 08:44:47.517775 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.517743 5055 scope.go:122] "RemoveContainer" containerID="d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c" Nov 02 08:44:47.518005 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:47.517981 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c\": container with ID starting with d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c not found: ID does not exist" containerID="d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c" Nov 02 08:44:47.518085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.518015 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c"} err="failed to get container status \"d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c\": rpc error: code = NotFound desc = could not find container \"d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c\": container with ID starting with d16ade075492a8f8170e97496eadf22157d28f8b5aa074ad11b1daea26889a4c not found: ID does not exist" Nov 02 08:44:47.650646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.650609 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd"] Nov 02 08:44:47.654585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:47.654558 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad4580-openshift-default-5f89696b85-74djd"] Nov 02 08:44:49.093661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:49.093623 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" path="/var/lib/kubelet/pods/695546c1-9771-4a9b-a2bd-23ef70aa73f5/volumes" Nov 02 08:44:51.293736 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.293694 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr"] Nov 02 08:44:51.294282 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.293936 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerName="discovery" containerID="cri-o://24bb2eefdf91b83d6a66e3b3ef3c49a787f5139bf2395980e99162d812de3dcb" gracePeriod=30 Nov 02 08:44:51.533819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.533776 5055 generic.go:356] "Generic (PLEG): container finished" podID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerID="24bb2eefdf91b83d6a66e3b3ef3c49a787f5139bf2395980e99162d812de3dcb" exitCode=0 Nov 02 08:44:51.534045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.533855 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" event={"ID":"2ad2561a-78e2-4044-8ced-a14cfad60342","Type":"ContainerDied","Data":"24bb2eefdf91b83d6a66e3b3ef3c49a787f5139bf2395980e99162d812de3dcb"} Nov 02 08:44:51.625570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.625538 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:44:51.727866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.727820 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.727882 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.727917 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.727965 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728188 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.728111 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.728241 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.728326 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap\") pod \"2ad2561a-78e2-4044-8ced-a14cfad60342\" (UID: \"2ad2561a-78e2-4044-8ced-a14cfad60342\") " Nov 02 08:44:51.728819 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.728786 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:51.730904 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.730871 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:51.731031 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.731011 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:51.731153 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.731124 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:51.731276 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.731255 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "kube-api-access-krtzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:51.731350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.731335 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:51.731415 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.731402 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs" pod "2ad2561a-78e2-4044-8ced-a14cfad60342" (UID: "2ad2561a-78e2-4044-8ced-a14cfad60342"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:51.830486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830389 5055 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2561a-78e2-4044-8ced-a14cfad60342-local-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:51.830812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830706 5055 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-ca-configmap\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:51.830812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830735 5055 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-kubeconfig\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:51.830812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830750 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krtzd\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-kube-api-access-krtzd\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:51.830812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830764 5055 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-cacerts\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:51.830812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830779 5055 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-csr-dns-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:51.830812 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:51.830792 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2561a-78e2-4044-8ced-a14cfad60342-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:52.542646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:52.542604 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" event={"ID":"2ad2561a-78e2-4044-8ced-a14cfad60342","Type":"ContainerDied","Data":"5156692234e12d0c378c143ef1e693cc39a9af9db74b682aa3f1335733b14f60"} Nov 02 08:44:52.542646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:52.542645 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" Nov 02 08:44:52.543164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:52.542662 5055 scope.go:122] "RemoveContainer" containerID="24bb2eefdf91b83d6a66e3b3ef3c49a787f5139bf2395980e99162d812de3dcb" Nov 02 08:44:52.746502 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:52.746464 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr"] Nov 02 08:44:52.749772 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:52.749742 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr"] Nov 02 08:44:53.092395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:53.092364 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" path="/var/lib/kubelet/pods/2ad2561a-78e2-4044-8ced-a14cfad60342/volumes" Nov 02 08:44:54.922129 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:44:54.922085 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:44:55.852983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:55.852940 5055 scope.go:122] "RemoveContainer" containerID="e7160460d7d9a122c1419c42ca5db20d9710eaae1621fad0ac37f1b0a73c55ef" Nov 02 08:44:55.873477 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:55.873448 5055 scope.go:122] "RemoveContainer" containerID="467c53126b5404a6acaa408bfbd03dc43ac62773085b4edc7196a49da258075b" Nov 02 08:44:56.576828 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:56.576782 5055 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5p6pr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:44:56.577323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:56.576878 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p6pr" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.138:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:44:57.078160 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078101 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb"] Nov 02 08:44:57.078684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078670 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerName="discovery" Nov 02 08:44:57.078731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078686 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerName="discovery" Nov 02 08:44:57.078731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078697 5055 cpu_manager.go:414] "containerMap: removing container" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" Nov 02 08:44:57.078731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078702 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" Nov 02 08:44:57.078731 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078727 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" containerName="revoke-keys" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078733 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" containerName="revoke-keys" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078741 5055 cpu_manager.go:414] "containerMap: removing container" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078748 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078823 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" containerName="revoke-keys" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078833 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078844 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad2561a-78e2-4044-8ced-a14cfad60342" containerName="discovery" Nov 02 08:44:57.078869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.078849 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="695546c1-9771-4a9b-a2bd-23ef70aa73f5" containerName="istio-proxy" Nov 02 08:44:57.084568 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.084537 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.134335 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.134304 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b"] Nov 02 08:44:57.138390 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.138367 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.177208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.177166 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:44:57.177208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.177187 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:44:57.177486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.177259 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:44:57.178914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.178886 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:44:57.184575 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.184536 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb"] Nov 02 08:44:57.195685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.195653 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.195869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.195701 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstfr\" (UniqueName: \"kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.195965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.195881 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.232419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.232384 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b"] Nov 02 08:44:57.296826 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.296786 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstfr\" (UniqueName: \"kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.297014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.296853 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.297014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.296897 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b889g\" (UniqueName: \"kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.297014 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.296993 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.297210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.297187 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.297268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.297252 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.297551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.297533 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.299718 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.299694 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.349921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.349840 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstfr\" (UniqueName: \"kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr\") pod \"maas-api-revoke-sub-3067018c-qlrkb\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.398737 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.398706 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b889g\" (UniqueName: \"kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.398923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.398795 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.398923 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.398901 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.399458 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.399436 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.400995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.400966 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:44:57.401283 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.401266 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.451899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.451869 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b889g\" (UniqueName: \"kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g\") pod \"maas-api-revoke-sub-04f04fec-7l59b\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.453690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.453660 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:44:57.772556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.772529 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb"] Nov 02 08:44:57.773675 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:44:57.773640 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e096a55_e02e_4752_8a7e_477b2321b407.slice/crio-2a03b01e1ae2e5d936e0e6e899d14f8c1d1e5198db492aecd83e6a87fe7a66e6 WatchSource:0}: Error finding container 2a03b01e1ae2e5d936e0e6e899d14f8c1d1e5198db492aecd83e6a87fe7a66e6: Status 404 returned error can't find the container with id 2a03b01e1ae2e5d936e0e6e899d14f8c1d1e5198db492aecd83e6a87fe7a66e6 Nov 02 08:44:57.875021 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:57.874989 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b"] Nov 02 08:44:57.875907 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:44:57.875879 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode978f39d_fcbf_4b92_a2b5_1eacdfe82fcb.slice/crio-b484cfedc9ceafd4932ee2d777421bc54e886f48d32f627fc4517f5c5038d65a WatchSource:0}: Error finding container b484cfedc9ceafd4932ee2d777421bc54e886f48d32f627fc4517f5c5038d65a: Status 404 returned error can't find the container with id b484cfedc9ceafd4932ee2d777421bc54e886f48d32f627fc4517f5c5038d65a Nov 02 08:44:58.600320 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:58.600272 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" event={"ID":"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb","Type":"ContainerStarted","Data":"697cc14ece152d75d188dc0d6290419c65973c818817bfea8bbbfab6d482f960"} Nov 02 08:44:58.600509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:58.600343 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" event={"ID":"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb","Type":"ContainerStarted","Data":"b484cfedc9ceafd4932ee2d777421bc54e886f48d32f627fc4517f5c5038d65a"} Nov 02 08:44:58.601993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:58.601967 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" event={"ID":"6e096a55-e02e-4752-8a7e-477b2321b407","Type":"ContainerStarted","Data":"5e9818c5846fd8454c1d980b47669566366856490becc107d36b26b221562b59"} Nov 02 08:44:58.602171 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:58.602017 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" event={"ID":"6e096a55-e02e-4752-8a7e-477b2321b407","Type":"ContainerStarted","Data":"2a03b01e1ae2e5d936e0e6e899d14f8c1d1e5198db492aecd83e6a87fe7a66e6"} Nov 02 08:44:58.708907 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:58.708850 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" podStartSLOduration=1.7088347339999999 podStartE2EDuration="1.708834734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:58.707550855 +0000 UTC m=+2345.798171275" watchObservedRunningTime="2026-11-02 08:44:58.708834734 +0000 UTC m=+2345.799455193" Nov 02 08:44:58.816193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:58.816117 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" podStartSLOduration=1.816104527 podStartE2EDuration="1.816104527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:58.814431634 +0000 UTC m=+2345.905052052" watchObservedRunningTime="2026-11-02 08:44:58.816104527 +0000 UTC m=+2345.906724950" Nov 02 08:44:59.610833 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:59.610798 5055 generic.go:356] "Generic (PLEG): container finished" podID="e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" containerID="697cc14ece152d75d188dc0d6290419c65973c818817bfea8bbbfab6d482f960" exitCode=0 Nov 02 08:44:59.611048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:59.610876 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" event={"ID":"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb","Type":"ContainerDied","Data":"697cc14ece152d75d188dc0d6290419c65973c818817bfea8bbbfab6d482f960"} Nov 02 08:44:59.612483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:59.612457 5055 generic.go:356] "Generic (PLEG): container finished" podID="6e096a55-e02e-4752-8a7e-477b2321b407" containerID="5e9818c5846fd8454c1d980b47669566366856490becc107d36b26b221562b59" exitCode=0 Nov 02 08:44:59.612600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:44:59.612484 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" event={"ID":"6e096a55-e02e-4752-8a7e-477b2321b407","Type":"ContainerDied","Data":"5e9818c5846fd8454c1d980b47669566366856490becc107d36b26b221562b59"} Nov 02 08:45:00.162329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.162292 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8"] Nov 02 08:45:00.166405 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.166377 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5"] Nov 02 08:45:00.166609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.166493 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.170675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.170648 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk"] Nov 02 08:45:00.170823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.170779 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.174848 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.174826 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.190382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.190348 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6"] Nov 02 08:45:00.194946 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.194918 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws"] Nov 02 08:45:00.195096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.195021 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.199560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.199533 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.287211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.279281 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8"] Nov 02 08:45:00.287211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.284898 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5"] Nov 02 08:45:00.291199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.291168 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk"] Nov 02 08:45:00.319232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.319169 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws"] Nov 02 08:45:00.325394 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.325356 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6"] Nov 02 08:45:00.339345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339315 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.339509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339368 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5qq\" (UniqueName: \"kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq\") pod \"maas-api-key-cleanup-29893485-8rfrk\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.339509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339402 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.339509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339482 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.339619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339539 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52pm\" (UniqueName: \"kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm\") pod \"maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.339619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339610 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893485-8rfrk\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.339693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339640 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.339693 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339670 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjkx\" (UniqueName: \"kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx\") pod \"maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.339808 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339794 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.339854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.339822 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhbr\" (UniqueName: \"kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.441018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.440925 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893485-8rfrk\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.441018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.440975 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.441018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441010 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjkx\" (UniqueName: \"kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx\") pod \"maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.441358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441044 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.441358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441086 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhbr\" (UniqueName: \"kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.441358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441239 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.441358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5qq\" (UniqueName: \"kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq\") pod \"maas-api-key-cleanup-29893485-8rfrk\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.441590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441361 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.441590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441392 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.441590 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441463 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52pm\" (UniqueName: \"kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm\") pod \"maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.441953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.441996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.441962 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893485-8rfrk\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.442033 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.442016 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.442194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.442173 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.442194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.442211 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.495192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.495153 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhbr\" (UniqueName: \"kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr\") pod \"maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.495381 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.495317 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc\") pod \"maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.495466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.495437 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjkx\" (UniqueName: \"kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx\") pod \"maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.496070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.495948 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5qq\" (UniqueName: \"kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq\") pod \"maas-api-key-cleanup-29893485-8rfrk\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.496438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.496415 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52pm\" (UniqueName: \"kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm\") pod \"maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.498901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.498883 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:00.511360 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.511321 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:00.519668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.519633 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:00.782540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.782318 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:00.791758 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.791516 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:00.895365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.895314 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:45:00.897256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.896993 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk"] Nov 02 08:45:00.910280 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.910249 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:45:00.943439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.943005 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws"] Nov 02 08:45:00.950197 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:45:00.945365 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66a5c42_9998_475c_ac1f_48c1f89a3d47.slice/crio-15f5ea9c6a08196e3c4afdc42951539f2b9a3490ee91cd66177c641bf8c2a27b WatchSource:0}: Error finding container 15f5ea9c6a08196e3c4afdc42951539f2b9a3490ee91cd66177c641bf8c2a27b: Status 404 returned error can't find the container with id 15f5ea9c6a08196e3c4afdc42951539f2b9a3490ee91cd66177c641bf8c2a27b Nov 02 08:45:00.950197 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:00.948491 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6"] Nov 02 08:45:01.049994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.049896 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g\" (UniqueName: \"kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g\") pod \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " Nov 02 08:45:01.049994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.049990 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca\") pod \"6e096a55-e02e-4752-8a7e-477b2321b407\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " Nov 02 08:45:01.050413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050091 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca\") pod \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " Nov 02 08:45:01.050413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050251 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca\") pod \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\" (UID: \"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb\") " Nov 02 08:45:01.050413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050315 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr\" (UniqueName: \"kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr\") pod \"6e096a55-e02e-4752-8a7e-477b2321b407\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " Nov 02 08:45:01.050413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050362 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca\") pod \"6e096a55-e02e-4752-8a7e-477b2321b407\" (UID: \"6e096a55-e02e-4752-8a7e-477b2321b407\") " Nov 02 08:45:01.050690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050409 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca" pod "6e096a55-e02e-4752-8a7e-477b2321b407" (UID: "6e096a55-e02e-4752-8a7e-477b2321b407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:01.050690 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050628 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca" pod "e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" (UID: "e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:01.050807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050765 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:01.050807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.050783 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:01.081264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.081219 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca" pod "e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" (UID: "e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:01.081809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.081690 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca" pod "6e096a55-e02e-4752-8a7e-477b2321b407" (UID: "6e096a55-e02e-4752-8a7e-477b2321b407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:01.081809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.081770 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr" pod "6e096a55-e02e-4752-8a7e-477b2321b407" (UID: "6e096a55-e02e-4752-8a7e-477b2321b407"). InnerVolumeSpecName "kube-api-access-xstfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:01.081809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.081792 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g" pod "e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" (UID: "e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb"). InnerVolumeSpecName "kube-api-access-b889g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:01.151609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.151571 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e096a55-e02e-4752-8a7e-477b2321b407-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:01.151609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.151606 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b889g\" (UniqueName: \"kubernetes.io/projected/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-kube-api-access-b889g\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:01.151609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.151616 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:01.151903 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.151624 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstfr\" (UniqueName: \"kubernetes.io/projected/6e096a55-e02e-4752-8a7e-477b2321b407-kube-api-access-xstfr\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:01.185562 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.185529 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8"] Nov 02 08:45:01.217424 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.215758 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5"] Nov 02 08:45:01.217424 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:45:01.215864 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6283bcd2_da4e_46b7_aa7f_7b6aef535cb4.slice/crio-91307d32e6af355c1923b01b9a395a1226c82e4077b8a99060f38847d79d6d71 WatchSource:0}: Error finding container 91307d32e6af355c1923b01b9a395a1226c82e4077b8a99060f38847d79d6d71: Status 404 returned error can't find the container with id 91307d32e6af355c1923b01b9a395a1226c82e4077b8a99060f38847d79d6d71 Nov 02 08:45:01.634722 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.634680 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" event={"ID":"6e096a55-e02e-4752-8a7e-477b2321b407","Type":"ContainerDied","Data":"2a03b01e1ae2e5d936e0e6e899d14f8c1d1e5198db492aecd83e6a87fe7a66e6"} Nov 02 08:45:01.634887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.634694 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb" Nov 02 08:45:01.634887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.634772 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a03b01e1ae2e5d936e0e6e899d14f8c1d1e5198db492aecd83e6a87fe7a66e6" Nov 02 08:45:01.636374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.636347 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerStarted","Data":"3e655f6e8cf3b94b234b12773f2920432a06bfafd31dcf222662a0d639f1a24c"} Nov 02 08:45:01.636501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.636398 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerStarted","Data":"d497230a913282622a05e1b400e91e901808a68f3d7d45ec3b02e9d17b23d361"} Nov 02 08:45:01.638083 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.638055 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerStarted","Data":"6e303f1bf7815894edb1deff5d03e98635db56eb5c5bb3c7614aaaab361fb749"} Nov 02 08:45:01.638243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.638101 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerStarted","Data":"4074b6f66074aca9fd27e7184666987824396b5f8dc86540794b204f85c67545"} Nov 02 08:45:01.639546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.639516 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerStarted","Data":"9e7ba379f0b89210bfd9b7a5be07d5e14ac6a5d0fbb02b384202479cf9ee2dcf"} Nov 02 08:45:01.639655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.639561 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerStarted","Data":"91307d32e6af355c1923b01b9a395a1226c82e4077b8a99060f38847d79d6d71"} Nov 02 08:45:01.641061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.641041 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerStarted","Data":"beb1b0999f0593309f1f66d19030f877daa38dc4522d83f60215803cb3bff074"} Nov 02 08:45:01.641180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.641081 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerStarted","Data":"88306c817aef0d8ff7d000f1cdc52e4865215a32216e31328baff7e4e13e58e1"} Nov 02 08:45:01.642432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.642386 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" Nov 02 08:45:01.642528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.642387 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b" event={"ID":"e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb","Type":"ContainerDied","Data":"b484cfedc9ceafd4932ee2d777421bc54e886f48d32f627fc4517f5c5038d65a"} Nov 02 08:45:01.642528 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.642485 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b484cfedc9ceafd4932ee2d777421bc54e886f48d32f627fc4517f5c5038d65a" Nov 02 08:45:01.643897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.643877 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerStarted","Data":"b1dae331d51fa157776185e007dcb4235e2605325cdc8b5543c903374a460991"} Nov 02 08:45:01.644000 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.643908 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerStarted","Data":"15f5ea9c6a08196e3c4afdc42951539f2b9a3490ee91cd66177c641bf8c2a27b"} Nov 02 08:45:01.743286 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.743227 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" podStartSLOduration=1.743212879 podStartE2EDuration="1.743212879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:01.741177051 +0000 UTC m=+2348.831797467" watchObservedRunningTime="2026-11-02 08:45:01.743212879 +0000 UTC m=+2348.833833319" Nov 02 08:45:01.849340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:01.849281 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" podStartSLOduration=1.849266327 podStartE2EDuration="1.849266327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:01.847165097 +0000 UTC m=+2348.937785541" watchObservedRunningTime="2026-11-02 08:45:01.849266327 +0000 UTC m=+2348.939886748" Nov 02 08:45:02.004199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.004116 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" podStartSLOduration=2.004102627 podStartE2EDuration="2.004102627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:02.002821407 +0000 UTC m=+2349.093441826" watchObservedRunningTime="2026-11-02 08:45:02.004102627 +0000 UTC m=+2349.094723046" Nov 02 08:45:02.069180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.069123 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" containerID="cri-o://fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb" gracePeriod=45 Nov 02 08:45:02.095514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.095469 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" containerID="cri-o://b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148" gracePeriod=45 Nov 02 08:45:02.121481 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.121426 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" podStartSLOduration=2.121411936 podStartE2EDuration="2.121411936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:02.117914675 +0000 UTC m=+2349.208535094" watchObservedRunningTime="2026-11-02 08:45:02.121411936 +0000 UTC m=+2349.212032388" Nov 02 08:45:02.226288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.226216 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" podStartSLOduration=2.226194335 podStartE2EDuration="2.226194335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:02.225392264 +0000 UTC m=+2349.316012685" watchObservedRunningTime="2026-11-02 08:45:02.226194335 +0000 UTC m=+2349.316814755" Nov 02 08:45:02.370945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.370915 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:45:02.439465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.439433 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:45:02.467222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467179 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs\") pod \"32bad2f0-4316-4b3e-9d90-9748e059792d\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " Nov 02 08:45:02.467419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467297 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache\") pod \"32bad2f0-4316-4b3e-9d90-9748e059792d\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " Nov 02 08:45:02.467419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467338 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home\") pod \"32bad2f0-4316-4b3e-9d90-9748e059792d\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " Nov 02 08:45:02.467419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467382 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm\") pod \"32bad2f0-4316-4b3e-9d90-9748e059792d\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " Nov 02 08:45:02.467600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467420 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir\") pod \"32bad2f0-4316-4b3e-9d90-9748e059792d\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " Nov 02 08:45:02.467600 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467508 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h\" (UniqueName: \"kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h\") pod \"32bad2f0-4316-4b3e-9d90-9748e059792d\" (UID: \"32bad2f0-4316-4b3e-9d90-9748e059792d\") " Nov 02 08:45:02.467779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467752 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache" pod "32bad2f0-4316-4b3e-9d90-9748e059792d" (UID: "32bad2f0-4316-4b3e-9d90-9748e059792d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.468030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.467752 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home" pod "32bad2f0-4316-4b3e-9d90-9748e059792d" (UID: "32bad2f0-4316-4b3e-9d90-9748e059792d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.468095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.468065 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir" pod "32bad2f0-4316-4b3e-9d90-9748e059792d" (UID: "32bad2f0-4316-4b3e-9d90-9748e059792d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.468237 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.468222 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.468298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.468239 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.468298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.468249 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.470551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.470517 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h" pod "32bad2f0-4316-4b3e-9d90-9748e059792d" (UID: "32bad2f0-4316-4b3e-9d90-9748e059792d"). InnerVolumeSpecName "kube-api-access-2s54h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:02.470551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.470542 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs" pod "32bad2f0-4316-4b3e-9d90-9748e059792d" (UID: "32bad2f0-4316-4b3e-9d90-9748e059792d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:02.470749 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.470553 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm" pod "32bad2f0-4316-4b3e-9d90-9748e059792d" (UID: "32bad2f0-4316-4b3e-9d90-9748e059792d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.568673 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.568630 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache\") pod \"2f512082-1172-433d-9043-181b9895b517\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " Nov 02 08:45:02.568879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.568726 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm\") pod \"2f512082-1172-433d-9043-181b9895b517\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " Nov 02 08:45:02.568879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.568809 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir\") pod \"2f512082-1172-433d-9043-181b9895b517\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " Nov 02 08:45:02.568879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.568858 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv\" (UniqueName: \"kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv\") pod \"2f512082-1172-433d-9043-181b9895b517\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " Nov 02 08:45:02.569042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.568926 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache" pod "2f512082-1172-433d-9043-181b9895b517" (UID: "2f512082-1172-433d-9043-181b9895b517"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.569042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.568939 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home\") pod \"2f512082-1172-433d-9043-181b9895b517\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " Nov 02 08:45:02.569203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569063 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs\") pod \"2f512082-1172-433d-9043-181b9895b517\" (UID: \"2f512082-1172-433d-9043-181b9895b517\") " Nov 02 08:45:02.569203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569101 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir" pod "2f512082-1172-433d-9043-181b9895b517" (UID: "2f512082-1172-433d-9043-181b9895b517"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.569314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home" pod "2f512082-1172-433d-9043-181b9895b517" (UID: "2f512082-1172-433d-9043-181b9895b517"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.569617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569592 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s54h\" (UniqueName: \"kubernetes.io/projected/32bad2f0-4316-4b3e-9d90-9748e059792d-kube-api-access-2s54h\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.569617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569619 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.569790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569630 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.569790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569644 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bad2f0-4316-4b3e-9d90-9748e059792d-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.569790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569652 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.569790 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.569660 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bad2f0-4316-4b3e-9d90-9748e059792d-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.570960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.570937 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm" pod "2f512082-1172-433d-9043-181b9895b517" (UID: "2f512082-1172-433d-9043-181b9895b517"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.571076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.571025 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv" pod "2f512082-1172-433d-9043-181b9895b517" (UID: "2f512082-1172-433d-9043-181b9895b517"). InnerVolumeSpecName "kube-api-access-g4flv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:02.571196 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.571176 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs" pod "2f512082-1172-433d-9043-181b9895b517" (UID: "2f512082-1172-433d-9043-181b9895b517"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:02.653449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.653408 5055 generic.go:356] "Generic (PLEG): container finished" podID="2f512082-1172-433d-9043-181b9895b517" containerID="b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148" exitCode=0 Nov 02 08:45:02.653676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.653472 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" Nov 02 08:45:02.653676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.653492 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" event={"ID":"2f512082-1172-433d-9043-181b9895b517","Type":"ContainerDied","Data":"b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148"} Nov 02 08:45:02.653676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.653549 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" event={"ID":"2f512082-1172-433d-9043-181b9895b517","Type":"ContainerDied","Data":"fb0f1dcd1704bf50bbedf818dbaeac5ca89a0b85157bbc3c9a0bdb02dd565aea"} Nov 02 08:45:02.653676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.653573 5055 scope.go:122] "RemoveContainer" containerID="b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148" Nov 02 08:45:02.655588 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.655568 5055 generic.go:356] "Generic (PLEG): container finished" podID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerID="fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb" exitCode=0 Nov 02 08:45:02.655687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.655638 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" event={"ID":"32bad2f0-4316-4b3e-9d90-9748e059792d","Type":"ContainerDied","Data":"fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb"} Nov 02 08:45:02.655687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.655661 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" event={"ID":"32bad2f0-4316-4b3e-9d90-9748e059792d","Type":"ContainerDied","Data":"174329f2ccb42bd23e3527c1adb9577a65d68856e7584a1eb1493935f040c33a"} Nov 02 08:45:02.655822 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.655702 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt" Nov 02 08:45:02.659452 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.659425 5055 generic.go:356] "Generic (PLEG): container finished" podID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerID="3e655f6e8cf3b94b234b12773f2920432a06bfafd31dcf222662a0d639f1a24c" exitCode=60 Nov 02 08:45:02.659589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.659512 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerDied","Data":"3e655f6e8cf3b94b234b12773f2920432a06bfafd31dcf222662a0d639f1a24c"} Nov 02 08:45:02.659825 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.659803 5055 scope.go:122] "RemoveContainer" containerID="3e655f6e8cf3b94b234b12773f2920432a06bfafd31dcf222662a0d639f1a24c" Nov 02 08:45:02.662235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.662210 5055 generic.go:356] "Generic (PLEG): container finished" podID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerID="6e303f1bf7815894edb1deff5d03e98635db56eb5c5bb3c7614aaaab361fb749" exitCode=60 Nov 02 08:45:02.669525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.662528 5055 scope.go:122] "RemoveContainer" containerID="6e303f1bf7815894edb1deff5d03e98635db56eb5c5bb3c7614aaaab361fb749" Nov 02 08:45:02.669525 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.666216 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerDied","Data":"6e303f1bf7815894edb1deff5d03e98635db56eb5c5bb3c7614aaaab361fb749"} Nov 02 08:45:02.671183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.671116 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f512082-1172-433d-9043-181b9895b517-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.671183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.671158 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4flv\" (UniqueName: \"kubernetes.io/projected/2f512082-1172-433d-9043-181b9895b517-kube-api-access-g4flv\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.671183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.671172 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f512082-1172-433d-9043-181b9895b517-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.672427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.672397 5055 generic.go:356] "Generic (PLEG): container finished" podID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerID="9e7ba379f0b89210bfd9b7a5be07d5e14ac6a5d0fbb02b384202479cf9ee2dcf" exitCode=60 Nov 02 08:45:02.672549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.672491 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerDied","Data":"9e7ba379f0b89210bfd9b7a5be07d5e14ac6a5d0fbb02b384202479cf9ee2dcf"} Nov 02 08:45:02.672770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.672730 5055 scope.go:122] "RemoveContainer" containerID="9e7ba379f0b89210bfd9b7a5be07d5e14ac6a5d0fbb02b384202479cf9ee2dcf" Nov 02 08:45:02.673234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.673211 5055 scope.go:122] "RemoveContainer" containerID="b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148" Nov 02 08:45:02.675268 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:02.675105 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148\": container with ID starting with b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148 not found: ID does not exist" containerID="b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148" Nov 02 08:45:02.675268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.675170 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148"} err="failed to get container status \"b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148\": rpc error: code = NotFound desc = could not find container \"b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148\": container with ID starting with b2c66b57116dd20be43383c83a1024e03ff5e8f2aceca2dce9ca805cc45d1148 not found: ID does not exist" Nov 02 08:45:02.675268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.675193 5055 scope.go:122] "RemoveContainer" containerID="fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb" Nov 02 08:45:02.676419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.676249 5055 generic.go:356] "Generic (PLEG): container finished" podID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerID="beb1b0999f0593309f1f66d19030f877daa38dc4522d83f60215803cb3bff074" exitCode=60 Nov 02 08:45:02.676419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.676316 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerDied","Data":"beb1b0999f0593309f1f66d19030f877daa38dc4522d83f60215803cb3bff074"} Nov 02 08:45:02.676550 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.676513 5055 scope.go:122] "RemoveContainer" containerID="beb1b0999f0593309f1f66d19030f877daa38dc4522d83f60215803cb3bff074" Nov 02 08:45:02.678270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.678247 5055 generic.go:356] "Generic (PLEG): container finished" podID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerID="b1dae331d51fa157776185e007dcb4235e2605325cdc8b5543c903374a460991" exitCode=60 Nov 02 08:45:02.678371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.678311 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerDied","Data":"b1dae331d51fa157776185e007dcb4235e2605325cdc8b5543c903374a460991"} Nov 02 08:45:02.678471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.678456 5055 scope.go:122] "RemoveContainer" containerID="b1dae331d51fa157776185e007dcb4235e2605325cdc8b5543c903374a460991" Nov 02 08:45:02.687240 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.687124 5055 scope.go:122] "RemoveContainer" containerID="fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb" Nov 02 08:45:02.688298 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:02.688232 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb\": container with ID starting with fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb not found: ID does not exist" containerID="fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb" Nov 02 08:45:02.688865 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:02.688271 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb"} err="failed to get container status \"fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb\": rpc error: code = NotFound desc = could not find container \"fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb\": container with ID starting with fe7f7d4d5665eddaad01bca8657a323cea43731995bc6aa96380aedb4c70cddb not found: ID does not exist" Nov 02 08:45:03.232420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.232384 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb"] Nov 02 08:45:03.236118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.236090 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb"] Nov 02 08:45:03.368596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.368501 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/simulator-w2-10d2fb-kserve-5f9b559c76-sk7lb" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:45:03.450155 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.450100 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt"] Nov 02 08:45:03.463044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.457076 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/premium-w2-10d2fb-kserve-5b5f8f6949-sl4qt"] Nov 02 08:45:03.691782 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.691743 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerStarted","Data":"db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d"} Nov 02 08:45:03.697067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.697033 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerStarted","Data":"0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140"} Nov 02 08:45:03.699660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.699628 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerStarted","Data":"9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92"} Nov 02 08:45:03.706817 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.706788 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerStarted","Data":"771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8"} Nov 02 08:45:03.707083 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.707051 5055 scope.go:122] "RemoveContainer" containerID="771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8" Nov 02 08:45:03.708118 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:03.708092 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5_odh-ai-gateway-infra(6283bcd2-da4e-46b7-aa7f-7b6aef535cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" Nov 02 08:45:03.708996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:03.708971 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerStarted","Data":"ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3"} Nov 02 08:45:04.502204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.502164 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:45:04.718312 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.718268 5055 generic.go:356] "Generic (PLEG): container finished" podID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerID="771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8" exitCode=60 Nov 02 08:45:04.718497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.718352 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerDied","Data":"771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8"} Nov 02 08:45:04.718497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.718421 5055 scope.go:122] "RemoveContainer" containerID="9e7ba379f0b89210bfd9b7a5be07d5e14ac6a5d0fbb02b384202479cf9ee2dcf" Nov 02 08:45:04.718795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.718773 5055 scope.go:122] "RemoveContainer" containerID="771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8" Nov 02 08:45:04.719168 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:04.719110 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5_odh-ai-gateway-infra(6283bcd2-da4e-46b7-aa7f-7b6aef535cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" Nov 02 08:45:04.720648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.720623 5055 generic.go:356] "Generic (PLEG): container finished" podID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerID="ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3" exitCode=60 Nov 02 08:45:04.720741 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.720712 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerDied","Data":"ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3"} Nov 02 08:45:04.720986 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.720969 5055 scope.go:122] "RemoveContainer" containerID="ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3" Nov 02 08:45:04.721350 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:04.721323 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893485-8rfrk_odh-ai-gateway-infra(961ed8e0-0b7e-48a9-be67-d7469a1249d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" Nov 02 08:45:04.722875 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.722851 5055 generic.go:356] "Generic (PLEG): container finished" podID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerID="db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d" exitCode=60 Nov 02 08:45:04.722980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.722939 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerDied","Data":"db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d"} Nov 02 08:45:04.723231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.723202 5055 scope.go:122] "RemoveContainer" containerID="db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d" Nov 02 08:45:04.723471 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:04.723451 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6_odh-ai-gateway-infra(b66a5c42-9998-475c-ac1f-48c1f89a3d47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" Nov 02 08:45:04.725406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.725388 5055 generic.go:356] "Generic (PLEG): container finished" podID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerID="0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140" exitCode=60 Nov 02 08:45:04.725509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.725412 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerDied","Data":"0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140"} Nov 02 08:45:04.725721 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.725698 5055 scope.go:122] "RemoveContainer" containerID="0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140" Nov 02 08:45:04.726026 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:04.725998 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8_odh-ai-gateway-infra(1852c99f-e2fd-47d3-bfff-ab07cc518026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" Nov 02 08:45:04.727552 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.727531 5055 generic.go:356] "Generic (PLEG): container finished" podID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerID="9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92" exitCode=60 Nov 02 08:45:04.727639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.727604 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerDied","Data":"9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92"} Nov 02 08:45:04.727857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.727840 5055 scope.go:122] "RemoveContainer" containerID="9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92" Nov 02 08:45:04.728121 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:04.728102 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws_odh-ai-gateway-infra(b5e2e023-779d-4c98-af31-bbd1098c4b99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" Nov 02 08:45:04.732849 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.732825 5055 scope.go:122] "RemoveContainer" containerID="beb1b0999f0593309f1f66d19030f877daa38dc4522d83f60215803cb3bff074" Nov 02 08:45:04.748617 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.748591 5055 scope.go:122] "RemoveContainer" containerID="b1dae331d51fa157776185e007dcb4235e2605325cdc8b5543c903374a460991" Nov 02 08:45:04.763565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.763537 5055 scope.go:122] "RemoveContainer" containerID="3e655f6e8cf3b94b234b12773f2920432a06bfafd31dcf222662a0d639f1a24c" Nov 02 08:45:04.780337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:04.780308 5055 scope.go:122] "RemoveContainer" containerID="6e303f1bf7815894edb1deff5d03e98635db56eb5c5bb3c7614aaaab361fb749" Nov 02 08:45:05.097252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.097135 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f512082-1172-433d-9043-181b9895b517" path="/var/lib/kubelet/pods/2f512082-1172-433d-9043-181b9895b517/volumes" Nov 02 08:45:05.097723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.097706 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" path="/var/lib/kubelet/pods/32bad2f0-4316-4b3e-9d90-9748e059792d/volumes" Nov 02 08:45:05.738453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.738422 5055 scope.go:122] "RemoveContainer" containerID="771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8" Nov 02 08:45:05.738971 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:05.738764 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5_odh-ai-gateway-infra(6283bcd2-da4e-46b7-aa7f-7b6aef535cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" Nov 02 08:45:05.740296 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.740273 5055 scope.go:122] "RemoveContainer" containerID="ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3" Nov 02 08:45:05.740609 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:05.740587 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893485-8rfrk_odh-ai-gateway-infra(961ed8e0-0b7e-48a9-be67-d7469a1249d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" Nov 02 08:45:05.742290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.742271 5055 scope.go:122] "RemoveContainer" containerID="db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d" Nov 02 08:45:05.742573 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:05.742542 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6_odh-ai-gateway-infra(b66a5c42-9998-475c-ac1f-48c1f89a3d47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" Nov 02 08:45:05.744194 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.744175 5055 scope.go:122] "RemoveContainer" containerID="0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140" Nov 02 08:45:05.744474 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:05.744444 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8_odh-ai-gateway-infra(1852c99f-e2fd-47d3-bfff-ab07cc518026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" Nov 02 08:45:05.745844 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:05.745828 5055 scope.go:122] "RemoveContainer" containerID="9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92" Nov 02 08:45:05.746067 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:05.746041 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws_odh-ai-gateway-infra(b5e2e023-779d-4c98-af31-bbd1098c4b99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" Nov 02 08:45:06.362120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.362073 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt"] Nov 02 08:45:06.363765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.363440 5055 cpu_manager.go:414] "containerMap: removing container" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" Nov 02 08:45:06.363765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.363471 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" Nov 02 08:45:06.364266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364239 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6e096a55-e02e-4752-8a7e-477b2321b407" containerName="revoke-keys" Nov 02 08:45:06.364366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364266 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e096a55-e02e-4752-8a7e-477b2321b407" containerName="revoke-keys" Nov 02 08:45:06.364366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364296 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" containerName="revoke-keys" Nov 02 08:45:06.364366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364304 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" containerName="revoke-keys" Nov 02 08:45:06.364366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364322 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" Nov 02 08:45:06.364366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364331 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" Nov 02 08:45:06.364517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364491 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e096a55-e02e-4752-8a7e-477b2321b407" containerName="revoke-keys" Nov 02 08:45:06.364517 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364509 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" containerName="revoke-keys" Nov 02 08:45:06.364585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364519 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f512082-1172-433d-9043-181b9895b517" containerName="main" Nov 02 08:45:06.364585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.364535 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bad2f0-4316-4b3e-9d90-9748e059792d" containerName="main" Nov 02 08:45:06.369802 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.369748 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.428670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.428633 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d51175\"/\"openshift-service-ca.crt\"" Nov 02 08:45:06.431324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.431211 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d51175\"/\"default-dockercfg-p5gd2\"" Nov 02 08:45:06.435715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.435683 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d51175\"/\"kube-root-ca.crt\"" Nov 02 08:45:06.436655 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.436612 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt"] Nov 02 08:45:06.522605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.522563 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d51175\"/\"ipp-route-d51175-kserve-self-signed-certs\"" Nov 02 08:45:06.536884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.536053 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.536884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.536204 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.536884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.536257 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w8ss\" (UniqueName: \"kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.536884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.536353 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.536884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.536440 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.536884 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.536481 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.637558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.637468 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.637558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.637523 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w8ss\" (UniqueName: \"kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.637774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.637579 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.637774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.637633 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.637774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.637653 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.637774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.637693 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.638090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.638062 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.638182 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.638116 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.638224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.638184 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.639947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.639921 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.640290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.640271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.691159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.691113 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w8ss\" (UniqueName: \"kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss\") pod \"ipp-route-d51175-kserve-745b4d669c-f22wt\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:06.738606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:06.738561 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:07.114208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:07.114162 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt"] Nov 02 08:45:07.114786 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:45:07.114753 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c91d1da_1f81_4e38_a90a_e150b8332978.slice/crio-b102ae1c2154d59af9b35eab5e9caec0cac350c16a3361cf8e755b31da4cd09f WatchSource:0}: Error finding container b102ae1c2154d59af9b35eab5e9caec0cac350c16a3361cf8e755b31da4cd09f: Status 404 returned error can't find the container with id b102ae1c2154d59af9b35eab5e9caec0cac350c16a3361cf8e755b31da4cd09f Nov 02 08:45:07.769447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:07.769403 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" event={"ID":"8c91d1da-1f81-4e38-a90a-e150b8332978","Type":"ContainerStarted","Data":"4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa"} Nov 02 08:45:07.769872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:07.769459 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" event={"ID":"8c91d1da-1f81-4e38-a90a-e150b8332978","Type":"ContainerStarted","Data":"b102ae1c2154d59af9b35eab5e9caec0cac350c16a3361cf8e755b31da4cd09f"} Nov 02 08:45:07.883003 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:07.882943 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" podStartSLOduration=1.8829296229999999 podStartE2EDuration="1.882929623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:07.880519298 +0000 UTC m=+2354.971139717" watchObservedRunningTime="2026-11-02 08:45:07.882929623 +0000 UTC m=+2354.973550042" Nov 02 08:45:08.738697 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:08.738656 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:10.997830 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:10.997791 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc"] Nov 02 08:45:10.998336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:10.998041 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" containerID="cri-o://375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f" gracePeriod=30 Nov 02 08:45:12.876460 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:12.876424 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb"] Nov 02 08:45:12.922701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:12.922667 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e6ca80-76tvb"] Nov 02 08:45:13.092547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:13.092510 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679b6d8a-8fd8-4764-be2d-edd33d55a2f1" path="/var/lib/kubelet/pods/679b6d8a-8fd8-4764-be2d-edd33d55a2f1/volumes" Nov 02 08:45:15.084915 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:15.084875 5055 scope.go:122] "RemoveContainer" containerID="db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d" Nov 02 08:45:15.085464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:15.084937 5055 scope.go:122] "RemoveContainer" containerID="ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3" Nov 02 08:45:15.085464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:15.085077 5055 scope.go:122] "RemoveContainer" containerID="0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140" Nov 02 08:45:15.833763 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:15.833719 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerStarted","Data":"82cb1622c25e90fb7d670a197f82aaa953cdc1bec3132c716185a85b1e66b6ab"} Nov 02 08:45:15.835887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:15.835855 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerStarted","Data":"28dec323f99752cd7486370b1ca3bb0f04ea3cf92d9bd5ae196dbf31d7a9437d"} Nov 02 08:45:15.837773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:15.837747 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerStarted","Data":"322d1eb73e297e818ef0fc23c956b3264dc855f610ea102b7200bca5b728b4b7"} Nov 02 08:45:16.276372 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.276336 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:45:16.398459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398373 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398459 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398436 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398478 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398517 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398649 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398735 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398782 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.398889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398842 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:16.398889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398877 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.399154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398882 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:16.399154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.398943 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert\") pod \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\" (UID: \"a5653ed7-4d7c-4f4a-9c24-3d881beeea94\") " Nov 02 08:45:16.399154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.399054 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:16.399154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.399105 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:16.399598 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.399533 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:16.400268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.400242 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.400268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.400272 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.400463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.400289 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.400463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.400303 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.400463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.400318 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.401319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.401283 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:45:16.401514 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.401488 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:16.401651 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.401632 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "kube-api-access-t8fg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:16.401879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.401843 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token" pod "a5653ed7-4d7c-4f4a-9c24-3d881beeea94" (UID: "a5653ed7-4d7c-4f4a-9c24-3d881beeea94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:16.502187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.501813 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.502187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.501874 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.502187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.501897 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.502187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.501918 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8fg4\" (UniqueName: \"kubernetes.io/projected/a5653ed7-4d7c-4f4a-9c24-3d881beeea94-kube-api-access-t8fg4\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:16.738810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.738770 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:16.751780 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.751747 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:16.846466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.846428 5055 generic.go:356] "Generic (PLEG): container finished" podID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerID="28dec323f99752cd7486370b1ca3bb0f04ea3cf92d9bd5ae196dbf31d7a9437d" exitCode=60 Nov 02 08:45:16.846663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.846500 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerDied","Data":"28dec323f99752cd7486370b1ca3bb0f04ea3cf92d9bd5ae196dbf31d7a9437d"} Nov 02 08:45:16.846663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.846562 5055 scope.go:122] "RemoveContainer" containerID="db2650de500e5acd8ca4104eeafde7defa90e1a64b2ca31fa1ed0520989d023d" Nov 02 08:45:16.846954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.846929 5055 scope.go:122] "RemoveContainer" containerID="28dec323f99752cd7486370b1ca3bb0f04ea3cf92d9bd5ae196dbf31d7a9437d" Nov 02 08:45:16.847313 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:16.847289 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6_odh-ai-gateway-infra(b66a5c42-9998-475c-ac1f-48c1f89a3d47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" Nov 02 08:45:16.848647 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.848628 5055 generic.go:356] "Generic (PLEG): container finished" podID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerID="322d1eb73e297e818ef0fc23c956b3264dc855f610ea102b7200bca5b728b4b7" exitCode=60 Nov 02 08:45:16.848733 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.848699 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerDied","Data":"322d1eb73e297e818ef0fc23c956b3264dc855f610ea102b7200bca5b728b4b7"} Nov 02 08:45:16.849046 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.849027 5055 scope.go:122] "RemoveContainer" containerID="322d1eb73e297e818ef0fc23c956b3264dc855f610ea102b7200bca5b728b4b7" Nov 02 08:45:16.849322 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:16.849301 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8_odh-ai-gateway-infra(1852c99f-e2fd-47d3-bfff-ab07cc518026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" Nov 02 08:45:16.850429 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.850409 5055 generic.go:356] "Generic (PLEG): container finished" podID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerID="375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f" exitCode=0 Nov 02 08:45:16.850534 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.850445 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" Nov 02 08:45:16.850534 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.850490 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" event={"ID":"a5653ed7-4d7c-4f4a-9c24-3d881beeea94","Type":"ContainerDied","Data":"375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f"} Nov 02 08:45:16.850628 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.850541 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc" event={"ID":"a5653ed7-4d7c-4f4a-9c24-3d881beeea94","Type":"ContainerDied","Data":"d077e2ca2d8872a333ac1279abc97af423ed9d056c29cd577cab7e3a6c3c2564"} Nov 02 08:45:16.852558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.852537 5055 generic.go:356] "Generic (PLEG): container finished" podID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerID="82cb1622c25e90fb7d670a197f82aaa953cdc1bec3132c716185a85b1e66b6ab" exitCode=60 Nov 02 08:45:16.852635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.852606 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerDied","Data":"82cb1622c25e90fb7d670a197f82aaa953cdc1bec3132c716185a85b1e66b6ab"} Nov 02 08:45:16.853001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.852983 5055 scope.go:122] "RemoveContainer" containerID="82cb1622c25e90fb7d670a197f82aaa953cdc1bec3132c716185a85b1e66b6ab" Nov 02 08:45:16.853340 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:16.853305 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893485-8rfrk_odh-ai-gateway-infra(961ed8e0-0b7e-48a9-be67-d7469a1249d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" Nov 02 08:45:16.866911 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.866778 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:45:16.867018 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.866924 5055 scope.go:122] "RemoveContainer" containerID="0d39806472c0560332ebb8325c1a4cc1771385f789d7cdc72b8b08dfccff4140" Nov 02 08:45:16.882964 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.882940 5055 scope.go:122] "RemoveContainer" containerID="375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f" Nov 02 08:45:16.892885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.892859 5055 scope.go:122] "RemoveContainer" containerID="6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec" Nov 02 08:45:16.902576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.902553 5055 scope.go:122] "RemoveContainer" containerID="375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f" Nov 02 08:45:16.902859 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:16.902836 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f\": container with ID starting with 375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f not found: ID does not exist" containerID="375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f" Nov 02 08:45:16.902921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.902867 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f"} err="failed to get container status \"375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f\": rpc error: code = NotFound desc = could not find container \"375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f\": container with ID starting with 375ea575ce404dcf4b6168484d4f57979f8a0c7ce584573449c8fd1a99852d6f not found: ID does not exist" Nov 02 08:45:16.902921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.902889 5055 scope.go:122] "RemoveContainer" containerID="6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec" Nov 02 08:45:16.903195 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:16.903175 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec\": container with ID starting with 6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec not found: ID does not exist" containerID="6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec" Nov 02 08:45:16.903270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.903201 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec"} err="failed to get container status \"6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec\": rpc error: code = NotFound desc = could not find container \"6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec\": container with ID starting with 6a19773fe885c27576527c5d6f01b19c7ae435fc2a78c7ba12305d98829e1cec not found: ID does not exist" Nov 02 08:45:16.903270 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.903218 5055 scope.go:122] "RemoveContainer" containerID="ec4b211923458ff707f96d01ffd46ffc2062a564db4659250582bc4d088bdbf3" Nov 02 08:45:16.952464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:16.952433 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk"] Nov 02 08:45:17.061420 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.061326 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8"] Nov 02 08:45:17.093980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.087519 5055 scope.go:122] "RemoveContainer" containerID="9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92" Nov 02 08:45:17.093980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.088897 5055 scope.go:122] "RemoveContainer" containerID="771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8" Nov 02 08:45:17.165349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.165318 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6"] Nov 02 08:45:17.444050 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.444016 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc"] Nov 02 08:45:17.447331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.447302 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-rmwmc"] Nov 02 08:45:17.867239 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.867209 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerStarted","Data":"fc53bcec96896427be77b4cdfa9fb06f44ba34256f126ea6097916d9f9586ade"} Nov 02 08:45:17.871131 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:17.871103 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerStarted","Data":"6807c72b2db1b4d72f1242652ee8a4a47f8d3310158e1989990048578c98c205"} Nov 02 08:45:18.056045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.056013 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:18.148553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.148524 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr\" (UniqueName: \"kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr\") pod \"1852c99f-e2fd-47d3-bfff-ab07cc518026\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " Nov 02 08:45:18.148743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.148684 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca\") pod \"1852c99f-e2fd-47d3-bfff-ab07cc518026\" (UID: \"1852c99f-e2fd-47d3-bfff-ab07cc518026\") " Nov 02 08:45:18.149044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.149020 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca" pod "1852c99f-e2fd-47d3-bfff-ab07cc518026" (UID: "1852c99f-e2fd-47d3-bfff-ab07cc518026"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:18.149463 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.149443 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1852c99f-e2fd-47d3-bfff-ab07cc518026-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:18.150762 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.150732 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr" pod "1852c99f-e2fd-47d3-bfff-ab07cc518026" (UID: "1852c99f-e2fd-47d3-bfff-ab07cc518026"). InnerVolumeSpecName "kube-api-access-qxhbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:18.197837 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.197810 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:18.201126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.201102 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:18.251509 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.251470 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhbr\" (UniqueName: \"kubernetes.io/projected/1852c99f-e2fd-47d3-bfff-ab07cc518026-kube-api-access-qxhbr\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:18.352872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.352833 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca\") pod \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " Nov 02 08:45:18.353055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.352891 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca\") pod \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " Nov 02 08:45:18.353055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.352923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq\" (UniqueName: \"kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq\") pod \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\" (UID: \"961ed8e0-0b7e-48a9-be67-d7469a1249d9\") " Nov 02 08:45:18.353055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.352980 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm\" (UniqueName: \"kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm\") pod \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\" (UID: \"b66a5c42-9998-475c-ac1f-48c1f89a3d47\") " Nov 02 08:45:18.353287 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.353258 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca" pod "961ed8e0-0b7e-48a9-be67-d7469a1249d9" (UID: "961ed8e0-0b7e-48a9-be67-d7469a1249d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:18.353467 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.353446 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca" pod "b66a5c42-9998-475c-ac1f-48c1f89a3d47" (UID: "b66a5c42-9998-475c-ac1f-48c1f89a3d47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:18.355224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.355201 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm" pod "b66a5c42-9998-475c-ac1f-48c1f89a3d47" (UID: "b66a5c42-9998-475c-ac1f-48c1f89a3d47"). InnerVolumeSpecName "kube-api-access-b52pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:18.355330 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.355298 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq" pod "961ed8e0-0b7e-48a9-be67-d7469a1249d9" (UID: "961ed8e0-0b7e-48a9-be67-d7469a1249d9"). InnerVolumeSpecName "kube-api-access-hd5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:18.454437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.454403 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961ed8e0-0b7e-48a9-be67-d7469a1249d9-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:18.454437 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.454435 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a5c42-9998-475c-ac1f-48c1f89a3d47-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:18.454868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.454446 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5qq\" (UniqueName: \"kubernetes.io/projected/961ed8e0-0b7e-48a9-be67-d7469a1249d9-kube-api-access-hd5qq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:18.454868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.454456 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b52pm\" (UniqueName: \"kubernetes.io/projected/b66a5c42-9998-475c-ac1f-48c1f89a3d47-kube-api-access-b52pm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:18.882728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.882642 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" Nov 02 08:45:18.882728 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.882678 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6" event={"ID":"b66a5c42-9998-475c-ac1f-48c1f89a3d47","Type":"ContainerDied","Data":"15f5ea9c6a08196e3c4afdc42951539f2b9a3490ee91cd66177c641bf8c2a27b"} Nov 02 08:45:18.882984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.882746 5055 scope.go:122] "RemoveContainer" containerID="28dec323f99752cd7486370b1ca3bb0f04ea3cf92d9bd5ae196dbf31d7a9437d" Nov 02 08:45:18.884316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.884298 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" Nov 02 08:45:18.884316 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.884303 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8" event={"ID":"1852c99f-e2fd-47d3-bfff-ab07cc518026","Type":"ContainerDied","Data":"d497230a913282622a05e1b400e91e901808a68f3d7d45ec3b02e9d17b23d361"} Nov 02 08:45:18.886636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.886607 5055 generic.go:356] "Generic (PLEG): container finished" podID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerID="fc53bcec96896427be77b4cdfa9fb06f44ba34256f126ea6097916d9f9586ade" exitCode=60 Nov 02 08:45:18.886781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.886685 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerDied","Data":"fc53bcec96896427be77b4cdfa9fb06f44ba34256f126ea6097916d9f9586ade"} Nov 02 08:45:18.887103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.887084 5055 scope.go:122] "RemoveContainer" containerID="fc53bcec96896427be77b4cdfa9fb06f44ba34256f126ea6097916d9f9586ade" Nov 02 08:45:18.887445 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:18.887409 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws_odh-ai-gateway-infra(b5e2e023-779d-4c98-af31-bbd1098c4b99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" Nov 02 08:45:18.888886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.888865 5055 generic.go:356] "Generic (PLEG): container finished" podID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerID="6807c72b2db1b4d72f1242652ee8a4a47f8d3310158e1989990048578c98c205" exitCode=60 Nov 02 08:45:18.888999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.888940 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerDied","Data":"6807c72b2db1b4d72f1242652ee8a4a47f8d3310158e1989990048578c98c205"} Nov 02 08:45:18.889338 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.889319 5055 scope.go:122] "RemoveContainer" containerID="6807c72b2db1b4d72f1242652ee8a4a47f8d3310158e1989990048578c98c205" Nov 02 08:45:18.889603 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:18.889584 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5_odh-ai-gateway-infra(6283bcd2-da4e-46b7-aa7f-7b6aef535cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" Nov 02 08:45:18.890462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.890442 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" Nov 02 08:45:18.890583 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.890475 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk" event={"ID":"961ed8e0-0b7e-48a9-be67-d7469a1249d9","Type":"ContainerDied","Data":"88306c817aef0d8ff7d000f1cdc52e4865215a32216e31328baff7e4e13e58e1"} Nov 02 08:45:18.895676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.895582 5055 scope.go:122] "RemoveContainer" containerID="322d1eb73e297e818ef0fc23c956b3264dc855f610ea102b7200bca5b728b4b7" Nov 02 08:45:18.906116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.906098 5055 scope.go:122] "RemoveContainer" containerID="9972ece0994760f6d45619c9d467141790693a6f96a0639ed9632a0b68e4cf92" Nov 02 08:45:18.920118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.920088 5055 scope.go:122] "RemoveContainer" containerID="771fdfef34642eafbb1e06ca29bb9f6b194ff0d3cf5856c2d421912022a7c9c8" Nov 02 08:45:18.935521 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:18.935496 5055 scope.go:122] "RemoveContainer" containerID="82cb1622c25e90fb7d670a197f82aaa953cdc1bec3132c716185a85b1e66b6ab" Nov 02 08:45:19.095300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.095255 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" path="/var/lib/kubelet/pods/a5653ed7-4d7c-4f4a-9c24-3d881beeea94/volumes" Nov 02 08:45:19.095750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.095698 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws"] Nov 02 08:45:19.144793 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.144709 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8"] Nov 02 08:45:19.147846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.147815 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-096685-29893485-75fp8"] Nov 02 08:45:19.320939 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.320909 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5"] Nov 02 08:45:19.374906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.374864 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk"] Nov 02 08:45:19.393349 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.393312 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8rfrk"] Nov 02 08:45:19.503816 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.503777 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6"] Nov 02 08:45:19.511395 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:19.511360 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d51175-29893485-7vhb6"] Nov 02 08:45:20.114099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.114045 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:20.119960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.119767 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:20.284556 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.284500 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc\") pod \"b5e2e023-779d-4c98-af31-bbd1098c4b99\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " Nov 02 08:45:20.284777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.284651 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx\" (UniqueName: \"kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx\") pod \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " Nov 02 08:45:20.284777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.284726 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca\") pod \"b5e2e023-779d-4c98-af31-bbd1098c4b99\" (UID: \"b5e2e023-779d-4c98-af31-bbd1098c4b99\") " Nov 02 08:45:20.284777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.284764 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca\") pod \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\" (UID: \"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4\") " Nov 02 08:45:20.285675 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.285608 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca" pod "6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" (UID: "6283bcd2-da4e-46b7-aa7f-7b6aef535cb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:20.286365 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.286296 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca" pod "b5e2e023-779d-4c98-af31-bbd1098c4b99" (UID: "b5e2e023-779d-4c98-af31-bbd1098c4b99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:20.306711 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.306668 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc" pod "b5e2e023-779d-4c98-af31-bbd1098c4b99" (UID: "b5e2e023-779d-4c98-af31-bbd1098c4b99"). InnerVolumeSpecName "kube-api-access-bw8dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:20.333120 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.333078 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx" pod "6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" (UID: "6283bcd2-da4e-46b7-aa7f-7b6aef535cb4"). InnerVolumeSpecName "kube-api-access-rzjkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:20.386754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.386710 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjkx\" (UniqueName: \"kubernetes.io/projected/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-kube-api-access-rzjkx\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:20.386754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.386756 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2e023-779d-4c98-af31-bbd1098c4b99-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:20.387010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.386774 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4-maas-api-service-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:20.387010 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.386788 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/b5e2e023-779d-4c98-af31-bbd1098c4b99-kube-api-access-bw8dc\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:20.914920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.914882 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" Nov 02 08:45:20.914920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.914903 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws" event={"ID":"b5e2e023-779d-4c98-af31-bbd1098c4b99","Type":"ContainerDied","Data":"4074b6f66074aca9fd27e7184666987824396b5f8dc86540794b204f85c67545"} Nov 02 08:45:20.915522 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.914976 5055 scope.go:122] "RemoveContainer" containerID="fc53bcec96896427be77b4cdfa9fb06f44ba34256f126ea6097916d9f9586ade" Nov 02 08:45:20.916886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.916856 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" event={"ID":"6283bcd2-da4e-46b7-aa7f-7b6aef535cb4","Type":"ContainerDied","Data":"91307d32e6af355c1923b01b9a395a1226c82e4077b8a99060f38847d79d6d71"} Nov 02 08:45:20.916886 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.916880 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5" Nov 02 08:45:20.929373 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:20.929348 5055 scope.go:122] "RemoveContainer" containerID="6807c72b2db1b4d72f1242652ee8a4a47f8d3310158e1989990048578c98c205" Nov 02 08:45:21.068989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.068957 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws"] Nov 02 08:45:21.092897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.092861 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" path="/var/lib/kubelet/pods/1852c99f-e2fd-47d3-bfff-ab07cc518026/volumes" Nov 02 08:45:21.093422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.093397 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" path="/var/lib/kubelet/pods/961ed8e0-0b7e-48a9-be67-d7469a1249d9/volumes" Nov 02 08:45:21.093874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.093853 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" path="/var/lib/kubelet/pods/b66a5c42-9998-475c-ac1f-48c1f89a3d47/volumes" Nov 02 08:45:21.114570 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.114536 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-10d2fb-29893485-l6sws"] Nov 02 08:45:21.170994 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.170909 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5"] Nov 02 08:45:21.174180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:21.174155 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d91f6f-29893485-6tqq5"] Nov 02 08:45:22.046667 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:22.046632 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845"] Nov 02 08:45:22.092723 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:22.092686 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc"] Nov 02 08:45:22.104202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:22.104018 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd8a9ed-2n845"] Nov 02 08:45:22.104202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:22.104087 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46290c08-9wqjc"] Nov 02 08:45:23.091870 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:23.091825 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" path="/var/lib/kubelet/pods/6283bcd2-da4e-46b7-aa7f-7b6aef535cb4/volumes" Nov 02 08:45:23.092520 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:23.092496 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79013d27-112c-46cf-93d0-a070b2d7ff15" path="/var/lib/kubelet/pods/79013d27-112c-46cf-93d0-a070b2d7ff15/volumes" Nov 02 08:45:23.092965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:23.092945 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fe1463-eeec-4d0f-b685-7af72fac9966" path="/var/lib/kubelet/pods/b4fe1463-eeec-4d0f-b685-7af72fac9966/volumes" Nov 02 08:45:23.093717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:23.093694 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" path="/var/lib/kubelet/pods/b5e2e023-779d-4c98-af31-bbd1098c4b99/volumes" Nov 02 08:45:24.044052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:24.044012 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql"] Nov 02 08:45:24.099414 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:24.099379 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17aca2a8-z92ql"] Nov 02 08:45:24.923772 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:24.923736 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:45:25.094471 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:25.094427 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b5a7c5-1bf4-4fac-bb12-960a88cbd748" path="/var/lib/kubelet/pods/17b5a7c5-1bf4-4fac-bb12-960a88cbd748/volumes" Nov 02 08:45:32.050483 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:32.050443 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6"] Nov 02 08:45:32.096717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:32.096674 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d55b990-dw7c6"] Nov 02 08:45:32.099605 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:32.099579 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4"] Nov 02 08:45:32.102028 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:32.102008 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5c6fe0-47qm4"] Nov 02 08:45:33.092084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:33.092047 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727c2ca1-0467-4b12-900b-86cab652c148" path="/var/lib/kubelet/pods/727c2ca1-0467-4b12-900b-86cab652c148/volumes" Nov 02 08:45:33.092674 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:33.092658 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2" path="/var/lib/kubelet/pods/a48c3a92-e8f0-4a15-a0c5-a36df9f9f5e2/volumes" Nov 02 08:45:42.932751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:42.932714 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:45:43.668073 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:43.668038 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx"] Nov 02 08:45:43.668322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:43.668285 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" containerID="cri-o://1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde" gracePeriod=30 Nov 02 08:45:48.957538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:48.957505 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:45:49.057821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.057782 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.057848 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.057987 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058018 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058052 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058063 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:49.058340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058158 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:49.058340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058174 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058262 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058309 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058362 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:49.058548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058378 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:49.058548 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058395 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo\") pod \"88c2b98a-18f5-4b2e-a54a-ce702a109380\" (UID: \"88c2b98a-18f5-4b2e-a54a-ce702a109380\") " Nov 02 08:45:49.058706 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058685 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:49.058921 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058903 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.059009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058924 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.059009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058939 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.059009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058952 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.059009 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.058966 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c2b98a-18f5-4b2e-a54a-ce702a109380-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.060445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.060417 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:49.060632 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.060610 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:45:49.060743 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.060725 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:49.060846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.060828 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq" pod "88c2b98a-18f5-4b2e-a54a-ce702a109380" (UID: "88c2b98a-18f5-4b2e-a54a-ce702a109380"). InnerVolumeSpecName "kube-api-access-c4vfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:49.153410 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.153371 5055 generic.go:356] "Generic (PLEG): container finished" podID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerID="1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde" exitCode=0 Nov 02 08:45:49.153608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.153429 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" event={"ID":"88c2b98a-18f5-4b2e-a54a-ce702a109380","Type":"ContainerDied","Data":"1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde"} Nov 02 08:45:49.153608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.153468 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" Nov 02 08:45:49.153608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.153491 5055 scope.go:122] "RemoveContainer" containerID="1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde" Nov 02 08:45:49.153608 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.153479 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx" event={"ID":"88c2b98a-18f5-4b2e-a54a-ce702a109380","Type":"ContainerDied","Data":"b71ec2f619b0dcf4ae1ec5371789081e5395dbb0ff8ff1d6e990f68ad2610604"} Nov 02 08:45:49.160585 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.160557 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.160709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.160588 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.160709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.160604 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vfq\" (UniqueName: \"kubernetes.io/projected/88c2b98a-18f5-4b2e-a54a-ce702a109380-kube-api-access-c4vfq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.160709 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.160618 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c2b98a-18f5-4b2e-a54a-ce702a109380-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:49.165750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.165718 5055 scope.go:122] "RemoveContainer" containerID="dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb" Nov 02 08:45:49.176300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.176278 5055 scope.go:122] "RemoveContainer" containerID="1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde" Nov 02 08:45:49.176623 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:49.176604 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde\": container with ID starting with 1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde not found: ID does not exist" containerID="1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde" Nov 02 08:45:49.176698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.176638 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde"} err="failed to get container status \"1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde\": rpc error: code = NotFound desc = could not find container \"1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde\": container with ID starting with 1acc237221566bc29917be28afa850c76198a17ec00def0394b282f5c4933fde not found: ID does not exist" Nov 02 08:45:49.176698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.176671 5055 scope.go:122] "RemoveContainer" containerID="dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb" Nov 02 08:45:49.176977 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:49.176958 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb\": container with ID starting with dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb not found: ID does not exist" containerID="dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb" Nov 02 08:45:49.177025 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.176985 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb"} err="failed to get container status \"dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb\": rpc error: code = NotFound desc = could not find container \"dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb\": container with ID starting with dbd57ca6a79e54c3d567b4c7fa24e8d4b5a292869eaa11b5a4de1ed8817b0cbb not found: ID does not exist" Nov 02 08:45:49.249175 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.249128 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p"] Nov 02 08:45:49.253419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.253389 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-10d2fb-qgn6p"] Nov 02 08:45:49.310900 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.310868 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx"] Nov 02 08:45:49.315051 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:49.315021 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-lskwx"] Nov 02 08:45:51.092729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:51.092696 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8c5900-0fbf-4735-a85c-6c91cc3aa95d" path="/var/lib/kubelet/pods/7d8c5900-0fbf-4735-a85c-6c91cc3aa95d/volumes" Nov 02 08:45:51.093217 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:51.093202 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" path="/var/lib/kubelet/pods/88c2b98a-18f5-4b2e-a54a-ce702a109380/volumes" Nov 02 08:45:53.829701 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:53.829671 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:45:53.830713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:53.830689 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:45:54.920675 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:45:54.920618 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:45:55.961587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:55.961552 5055 scope.go:122] "RemoveContainer" containerID="a24ba88a9e4994369c56496d82b2a6525e7b5e35c5633829bd1d9ed7a0d818fa" Nov 02 08:45:55.972770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:45:55.972739 5055 scope.go:122] "RemoveContainer" containerID="ad702550dfdf959db74c96a355202d2b88b2b21f1486ce7ca9c8fbb9c210e327" Nov 02 08:46:06.586941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.586902 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62"] Nov 02 08:46:06.587465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587449 5055 cpu_manager.go:414] "containerMap: removing container" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587465 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587479 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587484 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587492 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587497 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587505 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587511 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587521 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587526 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587535 5055 cpu_manager.go:414] "containerMap: removing container" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587539 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587546 5055 cpu_manager.go:414] "containerMap: removing container" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587551 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587558 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587563 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587571 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587575 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587583 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587587 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587600 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587604 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587616 5055 cpu_manager.go:414] "containerMap: removing container" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587621 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587633 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587637 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587643 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587647 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587654 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587659 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587734 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587741 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587747 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587754 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587761 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587768 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587773 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587780 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587786 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6283bcd2-da4e-46b7-aa7f-7b6aef535cb4" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587790 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587797 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587803 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c2b98a-18f5-4b2e-a54a-ce702a109380" containerName="istio-proxy" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587809 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587813 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:06.587801 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.587819 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:06.593941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.593918 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.642466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.642432 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:46:06.656118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.656084 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96c2\" (UniqueName: \"kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.656328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.656265 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.656465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.656447 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.688567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.688524 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:46:06.688567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.688524 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:46:06.688834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.688660 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:46:06.691742 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.691712 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62"] Nov 02 08:46:06.757350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.757317 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.757554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.757380 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96c2\" (UniqueName: \"kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.757554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.757464 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.757966 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.757944 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.759804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.759780 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.810547 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.810520 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96c2\" (UniqueName: \"kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:06.907659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:06.907569 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:07.275541 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:07.275513 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62"] Nov 02 08:46:07.278959 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:46:07.278929 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b55d31_88a9_4c2e_b034_600f1c30136f.slice/crio-608ff03e92ec73757a27a8b8640b45f5f93778dfa56cbad38ee52100f7417bf7 WatchSource:0}: Error finding container 608ff03e92ec73757a27a8b8640b45f5f93778dfa56cbad38ee52100f7417bf7: Status 404 returned error can't find the container with id 608ff03e92ec73757a27a8b8640b45f5f93778dfa56cbad38ee52100f7417bf7 Nov 02 08:46:07.301526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:07.301490 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" event={"ID":"15b55d31-88a9-4c2e-b034-600f1c30136f","Type":"ContainerStarted","Data":"608ff03e92ec73757a27a8b8640b45f5f93778dfa56cbad38ee52100f7417bf7"} Nov 02 08:46:08.309834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:08.309792 5055 generic.go:356] "Generic (PLEG): container finished" podID="15b55d31-88a9-4c2e-b034-600f1c30136f" containerID="26de72cba38db6a14d51503c37e2b3a79a54a80ade45c421053653c7236dab70" exitCode=0 Nov 02 08:46:08.310279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:08.309874 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" event={"ID":"15b55d31-88a9-4c2e-b034-600f1c30136f","Type":"ContainerDied","Data":"26de72cba38db6a14d51503c37e2b3a79a54a80ade45c421053653c7236dab70"} Nov 02 08:46:09.473210 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.473176 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:09.589944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.589188 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca\") pod \"15b55d31-88a9-4c2e-b034-600f1c30136f\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " Nov 02 08:46:09.589944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.589313 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2\" (UniqueName: \"kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2\") pod \"15b55d31-88a9-4c2e-b034-600f1c30136f\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " Nov 02 08:46:09.589944 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.589373 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca\") pod \"15b55d31-88a9-4c2e-b034-600f1c30136f\" (UID: \"15b55d31-88a9-4c2e-b034-600f1c30136f\") " Nov 02 08:46:09.590494 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.590461 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca" pod "15b55d31-88a9-4c2e-b034-600f1c30136f" (UID: "15b55d31-88a9-4c2e-b034-600f1c30136f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:09.593134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.593095 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca" pod "15b55d31-88a9-4c2e-b034-600f1c30136f" (UID: "15b55d31-88a9-4c2e-b034-600f1c30136f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:09.593729 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.593698 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2" pod "15b55d31-88a9-4c2e-b034-600f1c30136f" (UID: "15b55d31-88a9-4c2e-b034-600f1c30136f"). InnerVolumeSpecName "kube-api-access-s96c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:09.690581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.690548 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:09.690581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.690577 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b55d31-88a9-4c2e-b034-600f1c30136f-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:09.690581 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:09.690588 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96c2\" (UniqueName: \"kubernetes.io/projected/15b55d31-88a9-4c2e-b034-600f1c30136f-kube-api-access-s96c2\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:10.328861 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:10.328805 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" event={"ID":"15b55d31-88a9-4c2e-b034-600f1c30136f","Type":"ContainerDied","Data":"608ff03e92ec73757a27a8b8640b45f5f93778dfa56cbad38ee52100f7417bf7"} Nov 02 08:46:10.329072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:10.328882 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="608ff03e92ec73757a27a8b8640b45f5f93778dfa56cbad38ee52100f7417bf7" Nov 02 08:46:10.329072 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:10.328821 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62" Nov 02 08:46:11.799774 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:11.799730 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:46:20.185053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185007 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8"] Nov 02 08:46:20.185851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185827 5055 cpu_manager.go:414] "containerMap: removing container" podUID="15b55d31-88a9-4c2e-b034-600f1c30136f" containerName="revoke-keys" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185852 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b55d31-88a9-4c2e-b034-600f1c30136f" containerName="revoke-keys" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185879 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185887 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185902 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185910 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66a5c42-9998-475c-ac1f-48c1f89a3d47" containerName="cleanup" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185932 5055 cpu_manager.go:414] "containerMap: removing container" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:20.185954 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185940 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185974 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.185984 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.186130 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1852c99f-e2fd-47d3-bfff-ab07cc518026" containerName="cleanup" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.186172 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="961ed8e0-0b7e-48a9-be67-d7469a1249d9" containerName="cleanup" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.186183 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5653ed7-4d7c-4f4a-9c24-3d881beeea94" containerName="istio-proxy" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.186193 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e2e023-779d-4c98-af31-bbd1098c4b99" containerName="cleanup" Nov 02 08:46:20.186358 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.186206 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b55d31-88a9-4c2e-b034-600f1c30136f" containerName="revoke-keys" Nov 02 08:46:20.189990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.189964 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.284838 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.284804 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:46:20.285076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.284826 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:46:20.285076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.284826 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:46:20.285076 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.284881 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:46:20.288324 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.288299 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8"] Nov 02 08:46:20.316071 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.316039 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.316263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.316091 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.316263 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.316193 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qr7p\" (UniqueName: \"kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.417473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.417427 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.417659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.417508 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.417659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.417539 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qr7p\" (UniqueName: \"kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.418281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.418256 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.419888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.419868 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.470868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.470831 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qr7p\" (UniqueName: \"kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p\") pod \"maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.504472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.504430 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:20.877035 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:20.877003 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8"] Nov 02 08:46:20.878997 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:46:20.878730 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306fc1e3_d899_4c10_b6e4_f7bf9787d02e.slice/crio-d0f72c04fc5f97eb54dfc1d7723002095975f08cb0f2c90e298a4e69c0ecc9b2 WatchSource:0}: Error finding container d0f72c04fc5f97eb54dfc1d7723002095975f08cb0f2c90e298a4e69c0ecc9b2: Status 404 returned error can't find the container with id d0f72c04fc5f97eb54dfc1d7723002095975f08cb0f2c90e298a4e69c0ecc9b2 Nov 02 08:46:21.409030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:21.408989 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" event={"ID":"306fc1e3-d899-4c10-b6e4-f7bf9787d02e","Type":"ContainerStarted","Data":"a90023f7c75b94cf32f7aece6c1cc2678b02e76371fe11f74747be67aff2db92"} Nov 02 08:46:21.409475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:21.409085 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" event={"ID":"306fc1e3-d899-4c10-b6e4-f7bf9787d02e","Type":"ContainerStarted","Data":"d0f72c04fc5f97eb54dfc1d7723002095975f08cb0f2c90e298a4e69c0ecc9b2"} Nov 02 08:46:21.516901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:21.516838 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" podStartSLOduration=1.5168234200000001 podStartE2EDuration="1.51682342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:21.514789229 +0000 UTC m=+2428.605409648" watchObservedRunningTime="2026-11-02 08:46:21.51682342 +0000 UTC m=+2428.607443874" Nov 02 08:46:22.417958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:22.417909 5055 generic.go:356] "Generic (PLEG): container finished" podID="306fc1e3-d899-4c10-b6e4-f7bf9787d02e" containerID="a90023f7c75b94cf32f7aece6c1cc2678b02e76371fe11f74747be67aff2db92" exitCode=0 Nov 02 08:46:22.418423 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:22.417965 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" event={"ID":"306fc1e3-d899-4c10-b6e4-f7bf9787d02e","Type":"ContainerDied","Data":"a90023f7c75b94cf32f7aece6c1cc2678b02e76371fe11f74747be67aff2db92"} Nov 02 08:46:23.576645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.576618 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:23.657055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.657013 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca\") pod \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " Nov 02 08:46:23.657302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.657124 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p\" (UniqueName: \"kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p\") pod \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " Nov 02 08:46:23.657302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.657245 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca\") pod \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\" (UID: \"306fc1e3-d899-4c10-b6e4-f7bf9787d02e\") " Nov 02 08:46:23.657586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.657558 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca" pod "306fc1e3-d899-4c10-b6e4-f7bf9787d02e" (UID: "306fc1e3-d899-4c10-b6e4-f7bf9787d02e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:23.659110 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.659090 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca" pod "306fc1e3-d899-4c10-b6e4-f7bf9787d02e" (UID: "306fc1e3-d899-4c10-b6e4-f7bf9787d02e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:23.659213 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.659198 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p" pod "306fc1e3-d899-4c10-b6e4-f7bf9787d02e" (UID: "306fc1e3-d899-4c10-b6e4-f7bf9787d02e"). InnerVolumeSpecName "kube-api-access-4qr7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:23.759001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.758965 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qr7p\" (UniqueName: \"kubernetes.io/projected/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-kube-api-access-4qr7p\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:23.759001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.758995 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:23.759001 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:23.759005 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306fc1e3-d899-4c10-b6e4-f7bf9787d02e-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:24.435177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:24.435117 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" Nov 02 08:46:24.435177 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:24.435168 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8" event={"ID":"306fc1e3-d899-4c10-b6e4-f7bf9787d02e","Type":"ContainerDied","Data":"d0f72c04fc5f97eb54dfc1d7723002095975f08cb0f2c90e298a4e69c0ecc9b2"} Nov 02 08:46:24.435461 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:24.435220 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f72c04fc5f97eb54dfc1d7723002095975f08cb0f2c90e298a4e69c0ecc9b2" Nov 02 08:46:24.920795 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:46:24.920757 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:46:32.823686 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:32.823646 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62"] Nov 02 08:46:32.869531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:32.869494 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceb40-wlf62"] Nov 02 08:46:33.093180 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:33.093074 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b55d31-88a9-4c2e-b034-600f1c30136f" path="/var/lib/kubelet/pods/15b55d31-88a9-4c2e-b034-600f1c30136f/volumes" Nov 02 08:46:37.103993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.103959 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp"] Nov 02 08:46:37.104481 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.104467 5055 cpu_manager.go:414] "containerMap: removing container" podUID="306fc1e3-d899-4c10-b6e4-f7bf9787d02e" containerName="revoke-keys" Nov 02 08:46:37.104527 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.104482 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306fc1e3-d899-4c10-b6e4-f7bf9787d02e" containerName="revoke-keys" Nov 02 08:46:37.104576 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.104567 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="306fc1e3-d899-4c10-b6e4-f7bf9787d02e" containerName="revoke-keys" Nov 02 08:46:37.108231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.108196 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.156750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.156709 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:46:37.202744 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.202705 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:46:37.202951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.202772 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:46:37.203024 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.202952 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:46:37.205917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.205893 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp"] Nov 02 08:46:37.219154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.219107 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnft8\" (UniqueName: \"kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.219322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.219302 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.219438 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.219420 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.320572 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.320533 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.320781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.320629 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.320781 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.320734 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnft8\" (UniqueName: \"kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.321364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.321342 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.323002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.322982 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.374901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.374821 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnft8\" (UniqueName: \"kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8\") pod \"maas-api-revoke-sub-e11887a9-dr8tp\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.421813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.421776 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:37.789997 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:37.789964 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp"] Nov 02 08:46:37.791314 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:46:37.791289 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2ca0a7_15bc_4ff8_9586_99282ce41fd6.slice/crio-3689d85a51859c8c7cab5b6c197fc1a66c62d2a77f3e7f69756e7d437e2ef4c6 WatchSource:0}: Error finding container 3689d85a51859c8c7cab5b6c197fc1a66c62d2a77f3e7f69756e7d437e2ef4c6: Status 404 returned error can't find the container with id 3689d85a51859c8c7cab5b6c197fc1a66c62d2a77f3e7f69756e7d437e2ef4c6 Nov 02 08:46:38.538409 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:38.538365 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" event={"ID":"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6","Type":"ContainerStarted","Data":"fbc751e67d6fefa41e1617a0697ec9503925dd39b25c150c2e7bbbc2437559fe"} Nov 02 08:46:38.538874 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:38.538429 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" event={"ID":"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6","Type":"ContainerStarted","Data":"3689d85a51859c8c7cab5b6c197fc1a66c62d2a77f3e7f69756e7d437e2ef4c6"} Nov 02 08:46:39.479279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:39.479215 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" podStartSLOduration=2.479195035 podStartE2EDuration="2.479195035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:38.644333522 +0000 UTC m=+2445.734953940" watchObservedRunningTime="2026-11-02 08:46:39.479195035 +0000 UTC m=+2446.569815449" Nov 02 08:46:39.479659 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:39.479642 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl"] Nov 02 08:46:39.547369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:39.547330 5055 generic.go:356] "Generic (PLEG): container finished" podID="fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" containerID="fbc751e67d6fefa41e1617a0697ec9503925dd39b25c150c2e7bbbc2437559fe" exitCode=0 Nov 02 08:46:39.548097 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:39.547839 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" event={"ID":"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6","Type":"ContainerDied","Data":"fbc751e67d6fefa41e1617a0697ec9503925dd39b25c150c2e7bbbc2437559fe"} Nov 02 08:46:39.549663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:39.549629 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" containerID="cri-o://d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb" gracePeriod=30 Nov 02 08:46:40.234809 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.234766 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q"] Nov 02 08:46:40.241715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.241675 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.297238 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.297196 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q"] Nov 02 08:46:40.357033 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.356992 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.357243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.357218 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.357369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.357352 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c59t\" (UniqueName: \"kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.458526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.458490 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.458684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.458602 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c59t\" (UniqueName: \"kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.458684 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.458653 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.459306 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.459280 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.460972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.460947 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.512223 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.512134 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c59t\" (UniqueName: \"kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t\") pod \"maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.559454 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.559423 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:40.759960 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.759933 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:40.862369 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.862296 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca\") pod \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " Nov 02 08:46:40.862512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.862446 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8\" (UniqueName: \"kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8\") pod \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " Nov 02 08:46:40.862512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.862501 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca\") pod \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\" (UID: \"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6\") " Nov 02 08:46:40.862642 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.862614 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca" pod "fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" (UID: "fd2ca0a7-15bc-4ff8-9586-99282ce41fd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:40.862885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.862848 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:40.864656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.864629 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8" pod "fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" (UID: "fd2ca0a7-15bc-4ff8-9586-99282ce41fd6"). InnerVolumeSpecName "kube-api-access-dnft8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:40.864753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.864672 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca" pod "fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" (UID: "fd2ca0a7-15bc-4ff8-9586-99282ce41fd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:40.964119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.964088 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnft8\" (UniqueName: \"kubernetes.io/projected/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-kube-api-access-dnft8\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:40.964119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:40.964117 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:41.005166 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:46:41.005120 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c0135_4b4d_44b2_926d_2528336e0ee8.slice/crio-1f083bccea07a238a51d9f4be2ec97bddabf25bc2a29055ff01538ae5601c633 WatchSource:0}: Error finding container 1f083bccea07a238a51d9f4be2ec97bddabf25bc2a29055ff01538ae5601c633: Status 404 returned error can't find the container with id 1f083bccea07a238a51d9f4be2ec97bddabf25bc2a29055ff01538ae5601c633 Nov 02 08:46:41.006345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.005844 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q"] Nov 02 08:46:41.565222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.565193 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" Nov 02 08:46:41.565704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.565217 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp" event={"ID":"fd2ca0a7-15bc-4ff8-9586-99282ce41fd6","Type":"ContainerDied","Data":"3689d85a51859c8c7cab5b6c197fc1a66c62d2a77f3e7f69756e7d437e2ef4c6"} Nov 02 08:46:41.565704 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.565274 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3689d85a51859c8c7cab5b6c197fc1a66c62d2a77f3e7f69756e7d437e2ef4c6" Nov 02 08:46:41.567005 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.566977 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" event={"ID":"bb2c0135-4b4d-44b2-926d-2528336e0ee8","Type":"ContainerStarted","Data":"69816e975fc3c4fc356bf4b10b00c233220935c031086bc8cbbf5c6555e3c881"} Nov 02 08:46:41.567176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.567025 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" event={"ID":"bb2c0135-4b4d-44b2-926d-2528336e0ee8","Type":"ContainerStarted","Data":"1f083bccea07a238a51d9f4be2ec97bddabf25bc2a29055ff01538ae5601c633"} Nov 02 08:46:41.675945 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:41.675890 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" podStartSLOduration=1.675877978 podStartE2EDuration="1.675877978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:41.673553433 +0000 UTC m=+2448.764173851" watchObservedRunningTime="2026-11-02 08:46:41.675877978 +0000 UTC m=+2448.766498397" Nov 02 08:46:42.575020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:42.574977 5055 generic.go:356] "Generic (PLEG): container finished" podID="bb2c0135-4b4d-44b2-926d-2528336e0ee8" containerID="69816e975fc3c4fc356bf4b10b00c233220935c031086bc8cbbf5c6555e3c881" exitCode=0 Nov 02 08:46:42.575466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:42.575030 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" event={"ID":"bb2c0135-4b4d-44b2-926d-2528336e0ee8","Type":"ContainerDied","Data":"69816e975fc3c4fc356bf4b10b00c233220935c031086bc8cbbf5c6555e3c881"} Nov 02 08:46:43.355965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.355928 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:46:43.742919 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.742886 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:43.897968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.897926 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca\") pod \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " Nov 02 08:46:43.897968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.897969 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t\" (UniqueName: \"kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t\") pod \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " Nov 02 08:46:43.898252 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.898026 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca\") pod \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\" (UID: \"bb2c0135-4b4d-44b2-926d-2528336e0ee8\") " Nov 02 08:46:43.898425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.898392 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca" pod "bb2c0135-4b4d-44b2-926d-2528336e0ee8" (UID: "bb2c0135-4b4d-44b2-926d-2528336e0ee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:43.898582 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.898543 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:43.900199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.900124 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca" pod "bb2c0135-4b4d-44b2-926d-2528336e0ee8" (UID: "bb2c0135-4b4d-44b2-926d-2528336e0ee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:43.900309 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:43.900243 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t" pod "bb2c0135-4b4d-44b2-926d-2528336e0ee8" (UID: "bb2c0135-4b4d-44b2-926d-2528336e0ee8"). InnerVolumeSpecName "kube-api-access-2c59t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:44.000219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.000185 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c59t\" (UniqueName: \"kubernetes.io/projected/bb2c0135-4b4d-44b2-926d-2528336e0ee8-kube-api-access-2c59t\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:44.000219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.000220 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2c0135-4b4d-44b2-926d-2528336e0ee8-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:44.592952 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.592923 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" Nov 02 08:46:44.593123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.592953 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q" event={"ID":"bb2c0135-4b4d-44b2-926d-2528336e0ee8","Type":"ContainerDied","Data":"1f083bccea07a238a51d9f4be2ec97bddabf25bc2a29055ff01538ae5601c633"} Nov 02 08:46:44.593123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.593002 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f083bccea07a238a51d9f4be2ec97bddabf25bc2a29055ff01538ae5601c633" Nov 02 08:46:44.833432 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.833364 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:46:44.911867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.911831 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.911880 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.911903 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.911941 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.911962 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912163 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:46:44.912331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912132 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912273 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:46:44.912331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912288 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:46:44.912331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912298 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912330 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912389 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:46:44.912656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912526 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert\") pod \"bd96a9ac-a932-4d9e-914e-475702f42dd4\" (UID: \"bd96a9ac-a932-4d9e-914e-475702f42dd4\") " Nov 02 08:46:44.912979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.912947 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:44.913089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.913027 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:44.913089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.913050 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:44.913089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.913065 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:44.913089 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.913079 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:44.914696 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.914661 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:46:44.914883 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.914863 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "kube-api-access-b5m8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:44.914968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.914930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:46:44.915304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:44.915288 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token" pod "bd96a9ac-a932-4d9e-914e-475702f42dd4" (UID: "bd96a9ac-a932-4d9e-914e-475702f42dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:45.013958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.013885 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5m8b\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-kube-api-access-b5m8b\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:45.013958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.013944 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:45.013958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.013956 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96a9ac-a932-4d9e-914e-475702f42dd4-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:45.013958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.013965 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:45.013958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.013977 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96a9ac-a932-4d9e-914e-475702f42dd4-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:45.603256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.603221 5055 generic.go:356] "Generic (PLEG): container finished" podID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerID="d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb" exitCode=0 Nov 02 08:46:45.603445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.603282 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" Nov 02 08:46:45.603445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.603311 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" event={"ID":"bd96a9ac-a932-4d9e-914e-475702f42dd4","Type":"ContainerDied","Data":"d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb"} Nov 02 08:46:45.603445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.603372 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl" event={"ID":"bd96a9ac-a932-4d9e-914e-475702f42dd4","Type":"ContainerDied","Data":"1a911acc2f8c6eb4437e3562909e783e29194bbbddd0db300595ed0a2fa41be6"} Nov 02 08:46:45.603445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.603401 5055 scope.go:122] "RemoveContainer" containerID="d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb" Nov 02 08:46:45.613917 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.613897 5055 scope.go:122] "RemoveContainer" containerID="379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8" Nov 02 08:46:45.624496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.624477 5055 scope.go:122] "RemoveContainer" containerID="d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb" Nov 02 08:46:45.624780 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:46:45.624759 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb\": container with ID starting with d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb not found: ID does not exist" containerID="d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb" Nov 02 08:46:45.624844 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.624790 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb"} err="failed to get container status \"d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb\": rpc error: code = NotFound desc = could not find container \"d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb\": container with ID starting with d78044871eda2b7c285c690ad833be98423496d04204d4335bf3ff2b990346fb not found: ID does not exist" Nov 02 08:46:45.624844 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.624813 5055 scope.go:122] "RemoveContainer" containerID="379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8" Nov 02 08:46:45.625112 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:46:45.625092 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8\": container with ID starting with 379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8 not found: ID does not exist" containerID="379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8" Nov 02 08:46:45.625224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.625117 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8"} err="failed to get container status \"379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8\": rpc error: code = NotFound desc = could not find container \"379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8\": container with ID starting with 379543d695a8b751cd25cd75c5f00ebcf77f95e329ae7e18990cc5ef7c64d5d8 not found: ID does not exist" Nov 02 08:46:45.757810 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.757771 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl"] Nov 02 08:46:45.803765 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:45.803721 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceb40-openshift-default-67b68b47cc-8fqpl"] Nov 02 08:46:47.092798 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:47.092756 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" path="/var/lib/kubelet/pods/bd96a9ac-a932-4d9e-914e-475702f42dd4/volumes" Nov 02 08:46:54.920717 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:46:54.920675 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:46:56.087938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:56.087903 5055 scope.go:122] "RemoveContainer" containerID="0738ebacf46677795c3aff032f8ce2e3cd61fc548703bbe9c7546b47c66e4bae" Nov 02 08:46:56.099408 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:56.099383 5055 scope.go:122] "RemoveContainer" containerID="2cdff706f6df9f38d09f8a26c7877749298c48c78f1b1bee0f5f62b4d2d1316b" Nov 02 08:46:56.110611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:56.110582 5055 scope.go:122] "RemoveContainer" containerID="216a8595bc5c8dbad53aa5b78a657258140b70a8cedd03234d63e18215ab987d" Nov 02 08:46:56.122322 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:56.122296 5055 scope.go:122] "RemoveContainer" containerID="ad89f084385c95e41116b0f674feaad95f236f1d073313120c285b84699fe505" Nov 02 08:46:56.132216 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:56.132195 5055 scope.go:122] "RemoveContainer" containerID="498b15ba644acc9c560e60be1030635d41ab98c01749dada37647b372661456b" Nov 02 08:46:56.140823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:46:56.140800 5055 scope.go:122] "RemoveContainer" containerID="9c6282c71918019ecbced9bd8bd21a1abb6eb47a2af4586e5b07e335ae08a6c2" Nov 02 08:47:05.047843 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:05.047738 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c"] Nov 02 08:47:05.096351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:05.096307 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cd785f-dng9c"] Nov 02 08:47:06.419462 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:06.419425 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:47:07.091566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:07.091534 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548dafa7-36af-4778-80bd-6b89eea80a29" path="/var/lib/kubelet/pods/548dafa7-36af-4778-80bd-6b89eea80a29/volumes" Nov 02 08:47:11.911770 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:11.911734 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8"] Nov 02 08:47:11.958899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:11.958855 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d51175-g9xs8"] Nov 02 08:47:13.092700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:13.092663 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306fc1e3-d899-4c10-b6e4-f7bf9787d02e" path="/var/lib/kubelet/pods/306fc1e3-d899-4c10-b6e4-f7bf9787d02e/volumes" Nov 02 08:47:13.197981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:13.197940 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q"] Nov 02 08:47:13.243636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:13.243595 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d91f6f-tg82q"] Nov 02 08:47:15.092204 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:15.092165 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2c0135-4b4d-44b2-926d-2528336e0ee8" path="/var/lib/kubelet/pods/bb2c0135-4b4d-44b2-926d-2528336e0ee8/volumes" Nov 02 08:47:17.238086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:17.238046 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt"] Nov 02 08:47:19.626187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:19.626152 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2"] Nov 02 08:47:19.626626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:19.626409 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" containerID="cri-o://9ce4ac1a01829d966d79646f4298bae11e1248ce1d566fe95246a24c40d61374" gracePeriod=30 Nov 02 08:47:24.876453 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.876418 5055 generic.go:356] "Generic (PLEG): container finished" podID="058b922b-5389-4fc1-a032-1a49596e1f31" containerID="9ce4ac1a01829d966d79646f4298bae11e1248ce1d566fe95246a24c40d61374" exitCode=0 Nov 02 08:47:24.876864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.876490 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" event={"ID":"058b922b-5389-4fc1-a032-1a49596e1f31","Type":"ContainerDied","Data":"9ce4ac1a01829d966d79646f4298bae11e1248ce1d566fe95246a24c40d61374"} Nov 02 08:47:24.876864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.876570 5055 scope.go:122] "RemoveContainer" containerID="4d3a1fcc96bf01cb9e71941616f9f13cc285c1d6d2695b9e13085dabad67a3b9" Nov 02 08:47:24.899107 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.899081 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:47:24.920843 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:47:24.920807 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:47:24.981988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.981950 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982018 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982053 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982121 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982243 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982319 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982347 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982425 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982452 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert\") pod \"058b922b-5389-4fc1-a032-1a49596e1f31\" (UID: \"058b922b-5389-4fc1-a032-1a49596e1f31\") " Nov 02 08:47:24.982558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982488 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:24.982732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982542 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:24.982732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982647 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:24.982924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.982881 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:24.983055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.983031 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:24.983123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.983053 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:24.983123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.983063 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:24.983123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.983080 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:24.983123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.983095 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:24.984586 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.984560 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:24.984712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.984583 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "kube-api-access-nwzgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:24.984776 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.984729 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:24.984776 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:24.984742 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo" pod "058b922b-5389-4fc1-a032-1a49596e1f31" (UID: "058b922b-5389-4fc1-a032-1a49596e1f31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:47:25.084466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.084432 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:25.084466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.084463 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058b922b-5389-4fc1-a032-1a49596e1f31-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:25.084466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.084473 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzgg\" (UniqueName: \"kubernetes.io/projected/058b922b-5389-4fc1-a032-1a49596e1f31-kube-api-access-nwzgg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:25.084725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.084482 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058b922b-5389-4fc1-a032-1a49596e1f31-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:25.084725 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.084493 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058b922b-5389-4fc1-a032-1a49596e1f31-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:25.886052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.886015 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" event={"ID":"058b922b-5389-4fc1-a032-1a49596e1f31","Type":"ContainerDied","Data":"94daab2cede4386065b1b0b26168703ad3edb14acb86f7d1faaeffd0c56fe838"} Nov 02 08:47:25.886610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.886080 5055 scope.go:122] "RemoveContainer" containerID="9ce4ac1a01829d966d79646f4298bae11e1248ce1d566fe95246a24c40d61374" Nov 02 08:47:25.886610 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:25.886030 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2" Nov 02 08:47:26.186466 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:26.186369 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2"] Nov 02 08:47:26.189814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:26.189787 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d91f6f-openshift-default-5776c68949-5rjv2"] Nov 02 08:47:27.092680 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:27.092647 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" path="/var/lib/kubelet/pods/058b922b-5389-4fc1-a032-1a49596e1f31/volumes" Nov 02 08:47:29.210679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:29.210636 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:47:32.259771 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.259714 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" podUID="8c91d1da-1f81-4e38-a90a-e150b8332978" containerName="main" containerID="cri-o://4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa" gracePeriod=45 Nov 02 08:47:32.536487 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.536457 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:47:32.572706 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.572671 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss\" (UniqueName: \"kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss\") pod \"8c91d1da-1f81-4e38-a90a-e150b8332978\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " Nov 02 08:47:32.572920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.572715 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home\") pod \"8c91d1da-1f81-4e38-a90a-e150b8332978\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " Nov 02 08:47:32.572920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.572775 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs\") pod \"8c91d1da-1f81-4e38-a90a-e150b8332978\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " Nov 02 08:47:32.572920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.572838 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm\") pod \"8c91d1da-1f81-4e38-a90a-e150b8332978\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " Nov 02 08:47:32.572920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.572869 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir\") pod \"8c91d1da-1f81-4e38-a90a-e150b8332978\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " Nov 02 08:47:32.573202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.572997 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache\") pod \"8c91d1da-1f81-4e38-a90a-e150b8332978\" (UID: \"8c91d1da-1f81-4e38-a90a-e150b8332978\") " Nov 02 08:47:32.573202 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.573046 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home" pod "8c91d1da-1f81-4e38-a90a-e150b8332978" (UID: "8c91d1da-1f81-4e38-a90a-e150b8332978"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:32.573317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.573235 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir" pod "8c91d1da-1f81-4e38-a90a-e150b8332978" (UID: "8c91d1da-1f81-4e38-a90a-e150b8332978"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:32.573374 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.573358 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache" pod "8c91d1da-1f81-4e38-a90a-e150b8332978" (UID: "8c91d1da-1f81-4e38-a90a-e150b8332978"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:32.573497 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.573482 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:32.573557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.573500 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:32.573557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.573513 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:32.575250 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.575219 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm" pod "8c91d1da-1f81-4e38-a90a-e150b8332978" (UID: "8c91d1da-1f81-4e38-a90a-e150b8332978"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:47:32.575465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.575444 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs" pod "8c91d1da-1f81-4e38-a90a-e150b8332978" (UID: "8c91d1da-1f81-4e38-a90a-e150b8332978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:47:32.575531 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.575510 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss" pod "8c91d1da-1f81-4e38-a90a-e150b8332978" (UID: "8c91d1da-1f81-4e38-a90a-e150b8332978"). InnerVolumeSpecName "kube-api-access-2w8ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:32.674710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.674672 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w8ss\" (UniqueName: \"kubernetes.io/projected/8c91d1da-1f81-4e38-a90a-e150b8332978-kube-api-access-2w8ss\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:32.674710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.674707 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c91d1da-1f81-4e38-a90a-e150b8332978-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:32.674930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.674721 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c91d1da-1f81-4e38-a90a-e150b8332978-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:32.936078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.935993 5055 generic.go:356] "Generic (PLEG): container finished" podID="8c91d1da-1f81-4e38-a90a-e150b8332978" containerID="4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa" exitCode=0 Nov 02 08:47:32.936078 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.936042 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" Nov 02 08:47:32.936304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.936037 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" event={"ID":"8c91d1da-1f81-4e38-a90a-e150b8332978","Type":"ContainerDied","Data":"4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa"} Nov 02 08:47:32.936304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.936173 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt" event={"ID":"8c91d1da-1f81-4e38-a90a-e150b8332978","Type":"ContainerDied","Data":"b102ae1c2154d59af9b35eab5e9caec0cac350c16a3361cf8e755b31da4cd09f"} Nov 02 08:47:32.936304 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.936202 5055 scope.go:122] "RemoveContainer" containerID="4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa" Nov 02 08:47:32.946889 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.946870 5055 scope.go:122] "RemoveContainer" containerID="4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa" Nov 02 08:47:32.947165 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:47:32.947128 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa\": container with ID starting with 4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa not found: ID does not exist" containerID="4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa" Nov 02 08:47:32.947234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:32.947169 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa"} err="failed to get container status \"4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa\": rpc error: code = NotFound desc = could not find container \"4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa\": container with ID starting with 4728f520cc18a2f1fc6d2405c218d8b835da0b11d58fd9e49a0a29c291599aaa not found: ID does not exist" Nov 02 08:47:33.094115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:33.094082 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt"] Nov 02 08:47:33.094115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:33.094119 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d51175/ipp-route-d51175-kserve-745b4d669c-f22wt"] Nov 02 08:47:35.091457 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:35.091417 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c91d1da-1f81-4e38-a90a-e150b8332978" path="/var/lib/kubelet/pods/8c91d1da-1f81-4e38-a90a-e150b8332978/volumes" Nov 02 08:47:54.920821 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:47:54.920776 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:47:56.198042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:56.198003 5055 scope.go:122] "RemoveContainer" containerID="b0eb58cf2eea58d0368d12c7110ebbc55c357c11fb78e911c409f29d019469be" Nov 02 08:47:57.164925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:57.164880 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:47:58.047687 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:58.047649 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4"] Nov 02 08:47:58.093988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:58.093946 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp"] Nov 02 08:47:58.096974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:58.096943 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471ca7bd-xbjp4"] Nov 02 08:47:58.100183 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:58.100159 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac608733-xpntp"] Nov 02 08:47:59.092984 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:59.092939 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8" path="/var/lib/kubelet/pods/1cf4bce4-c1d2-4615-b5ab-8aa91eae66f8/volumes" Nov 02 08:47:59.093537 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:47:59.093423 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bca7db-8bfd-40a7-97c8-823dcb58ffda" path="/var/lib/kubelet/pods/61bca7db-8bfd-40a7-97c8-823dcb58ffda/volumes" Nov 02 08:48:01.095847 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.095809 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h"] Nov 02 08:48:01.096337 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096323 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8c91d1da-1f81-4e38-a90a-e150b8332978" containerName="main" Nov 02 08:48:01.096383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096338 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c91d1da-1f81-4e38-a90a-e150b8332978" containerName="main" Nov 02 08:48:01.096383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096350 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" Nov 02 08:48:01.096383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096356 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" Nov 02 08:48:01.096383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096368 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bb2c0135-4b4d-44b2-926d-2528336e0ee8" containerName="revoke-keys" Nov 02 08:48:01.096383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096373 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2c0135-4b4d-44b2-926d-2528336e0ee8" containerName="revoke-keys" Nov 02 08:48:01.096383 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096382 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096387 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096408 5055 cpu_manager.go:414] "containerMap: removing container" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096413 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096422 5055 cpu_manager.go:414] "containerMap: removing container" podUID="fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" containerName="revoke-keys" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096426 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" containerName="revoke-keys" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096503 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" containerName="revoke-keys" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096516 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096523 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c91d1da-1f81-4e38-a90a-e150b8332978" containerName="main" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096529 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096535 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2c0135-4b4d-44b2-926d-2528336e0ee8" containerName="revoke-keys" Nov 02 08:48:01.096574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.096542 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd96a9ac-a932-4d9e-914e-475702f42dd4" containerName="istio-proxy" Nov 02 08:48:01.101748 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.101718 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.113496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.113466 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.113660 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.113582 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsth\" (UniqueName: \"kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.113823 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.113794 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.150444 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.150411 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:48:01.150623 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.150454 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:48:01.196604 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.196571 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:48:01.196826 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.196734 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:48:01.201126 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.201092 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h"] Nov 02 08:48:01.215265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.215233 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.215436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.215280 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.215436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.215387 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsth\" (UniqueName: \"kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.215968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.215941 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.217792 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.217772 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.269400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.269365 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsth\" (UniqueName: \"kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.415928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.415833 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:01.790066 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:01.790040 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h"] Nov 02 08:48:01.792310 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:48:01.792280 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bbb21d_41d6_4522_8e72_1b4c0b057611.slice/crio-e39c14091b32aaca1b9401c472ce2bd35d1fd1a4b6a20dfb431a7ae32073f5a7 WatchSource:0}: Error finding container e39c14091b32aaca1b9401c472ce2bd35d1fd1a4b6a20dfb431a7ae32073f5a7: Status 404 returned error can't find the container with id e39c14091b32aaca1b9401c472ce2bd35d1fd1a4b6a20dfb431a7ae32073f5a7 Nov 02 08:48:02.154755 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:02.154721 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" event={"ID":"29bbb21d-41d6-4522-8e72-1b4c0b057611","Type":"ContainerStarted","Data":"298f45a8d112fde6b61388ca32f1fb01f57eb67c9a6259ba265fe23b21926d1e"} Nov 02 08:48:02.155215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:02.154778 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" event={"ID":"29bbb21d-41d6-4522-8e72-1b4c0b057611","Type":"ContainerStarted","Data":"e39c14091b32aaca1b9401c472ce2bd35d1fd1a4b6a20dfb431a7ae32073f5a7"} Nov 02 08:48:02.261294 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:02.261236 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" podStartSLOduration=1.261221136 podStartE2EDuration="1.261221136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:48:02.260402513 +0000 UTC m=+2529.351022931" watchObservedRunningTime="2026-11-02 08:48:02.261221136 +0000 UTC m=+2529.351841555" Nov 02 08:48:03.163996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:03.163954 5055 generic.go:356] "Generic (PLEG): container finished" podID="29bbb21d-41d6-4522-8e72-1b4c0b057611" containerID="298f45a8d112fde6b61388ca32f1fb01f57eb67c9a6259ba265fe23b21926d1e" exitCode=0 Nov 02 08:48:03.164501 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:03.164062 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" event={"ID":"29bbb21d-41d6-4522-8e72-1b4c0b057611","Type":"ContainerDied","Data":"298f45a8d112fde6b61388ca32f1fb01f57eb67c9a6259ba265fe23b21926d1e"} Nov 02 08:48:04.327643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.327616 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:04.352664 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.352628 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca\") pod \"29bbb21d-41d6-4522-8e72-1b4c0b057611\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " Nov 02 08:48:04.352872 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.352714 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca\") pod \"29bbb21d-41d6-4522-8e72-1b4c0b057611\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " Nov 02 08:48:04.352958 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.352878 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth\" (UniqueName: \"kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth\") pod \"29bbb21d-41d6-4522-8e72-1b4c0b057611\" (UID: \"29bbb21d-41d6-4522-8e72-1b4c0b057611\") " Nov 02 08:48:04.353045 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.353027 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca" pod "29bbb21d-41d6-4522-8e72-1b4c0b057611" (UID: "29bbb21d-41d6-4522-8e72-1b4c0b057611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:04.353402 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.353376 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:04.354854 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.354822 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca" pod "29bbb21d-41d6-4522-8e72-1b4c0b057611" (UID: "29bbb21d-41d6-4522-8e72-1b4c0b057611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:48:04.354992 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.354975 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth" pod "29bbb21d-41d6-4522-8e72-1b4c0b057611" (UID: "29bbb21d-41d6-4522-8e72-1b4c0b057611"). InnerVolumeSpecName "kube-api-access-zrsth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:04.454226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.454191 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsth\" (UniqueName: \"kubernetes.io/projected/29bbb21d-41d6-4522-8e72-1b4c0b057611-kube-api-access-zrsth\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:04.454226 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:04.454227 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bbb21d-41d6-4522-8e72-1b4c0b057611-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:05.179015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:05.178982 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" Nov 02 08:48:05.179224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:05.179016 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h" event={"ID":"29bbb21d-41d6-4522-8e72-1b4c0b057611","Type":"ContainerDied","Data":"e39c14091b32aaca1b9401c472ce2bd35d1fd1a4b6a20dfb431a7ae32073f5a7"} Nov 02 08:48:05.179224 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:05.179069 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39c14091b32aaca1b9401c472ce2bd35d1fd1a4b6a20dfb431a7ae32073f5a7" Nov 02 08:48:19.568545 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:19.568504 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv"] Nov 02 08:48:19.569095 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:19.568723 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" containerID="cri-o://daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e" gracePeriod=30 Nov 02 08:48:24.061866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.061829 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm"] Nov 02 08:48:24.062449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.062431 5055 cpu_manager.go:414] "containerMap: removing container" podUID="29bbb21d-41d6-4522-8e72-1b4c0b057611" containerName="revoke-keys" Nov 02 08:48:24.062496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.062450 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bbb21d-41d6-4522-8e72-1b4c0b057611" containerName="revoke-keys" Nov 02 08:48:24.062496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.062486 5055 cpu_manager.go:414] "containerMap: removing container" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" Nov 02 08:48:24.062496 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.062492 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" Nov 02 08:48:24.062602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.062578 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="058b922b-5389-4fc1-a032-1a49596e1f31" containerName="istio-proxy" Nov 02 08:48:24.062602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.062587 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bbb21d-41d6-4522-8e72-1b4c0b057611" containerName="revoke-keys" Nov 02 08:48:24.065895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.065875 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099323 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099282 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099410 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099437 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099551 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099469 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099568 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099640 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099746 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.099990 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.099968 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.100042 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.100014 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ljh\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.174846 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.174802 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm"] Nov 02 08:48:24.186378 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.186344 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bbb1eb-openshift-default-dockercfg-2dr5x\"" Nov 02 08:48:24.201465 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201432 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.201657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201526 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.201657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201551 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.201657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201581 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.201657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201606 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.201657 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201627 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.201969 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.201673 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202055 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202273 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202167 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ljh\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202399 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202374 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202463 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202652 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202822 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.202863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.202844 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.205123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.205046 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.213099 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.211097 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.265745 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.265695 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.267857 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.267822 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ljh\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh\") pod \"e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.379308 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.379223 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:24.822862 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.822831 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:48:24.851409 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:48:24.851331 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25f01e3_98c1_46ea_a1c8_236f011cc3a5.slice/crio-475c8ace2634abe4804a687788dddefda337b0ede33a4523856c1a40e3faeb6b WatchSource:0}: Error finding container 475c8ace2634abe4804a687788dddefda337b0ede33a4523856c1a40e3faeb6b: Status 404 returned error can't find the container with id 475c8ace2634abe4804a687788dddefda337b0ede33a4523856c1a40e3faeb6b Nov 02 08:48:24.851565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.851448 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm"] Nov 02 08:48:24.853924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.853890 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:48:24.854061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.853961 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:48:24.854061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.853993 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:48:24.908961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.908931 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.908970 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909004 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909038 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909084 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909163 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909302 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909341 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909335 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.909426 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909378 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket\") pod \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\" (UID: \"4381ccf4-b9f8-407c-a053-1a6ac3a352fd\") " Nov 02 08:48:24.909800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909775 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:24.910015 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.909993 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.910104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.910015 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.910222 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.910098 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:24.910451 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.910429 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.911520 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.911489 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:48:24.911779 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.911760 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.911924 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.911905 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:24.912215 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:24.912126 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t" pod "4381ccf4-b9f8-407c-a053-1a6ac3a352fd" (UID: "4381ccf4-b9f8-407c-a053-1a6ac3a352fd"). InnerVolumeSpecName "kube-api-access-69h2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:24.923428 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:48:24.923391 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:48:25.010991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.010941 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.010991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.010988 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.010991 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.010999 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.011392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.011008 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.011392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.011019 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.011392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.011027 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.011392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.011036 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69h2t\" (UniqueName: \"kubernetes.io/projected/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-kube-api-access-69h2t\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.011392 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.011047 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4381ccf4-b9f8-407c-a053-1a6ac3a352fd-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.337269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.337158 5055 generic.go:356] "Generic (PLEG): container finished" podID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerID="daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e" exitCode=0 Nov 02 08:48:25.337269 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.337234 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" event={"ID":"4381ccf4-b9f8-407c-a053-1a6ac3a352fd","Type":"ContainerDied","Data":"daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e"} Nov 02 08:48:25.337925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.337269 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" Nov 02 08:48:25.337925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.337310 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv" event={"ID":"4381ccf4-b9f8-407c-a053-1a6ac3a352fd","Type":"ContainerDied","Data":"ea1fac619e93332928e47983a8114903c9b1ec2e22903fd313278b02b49f519b"} Nov 02 08:48:25.337925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.337345 5055 scope.go:122] "RemoveContainer" containerID="daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e" Nov 02 08:48:25.340085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.339953 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" event={"ID":"c25f01e3-98c1-46ea-a1c8-236f011cc3a5","Type":"ContainerStarted","Data":"1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5"} Nov 02 08:48:25.340085 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.340034 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" event={"ID":"c25f01e3-98c1-46ea-a1c8-236f011cc3a5","Type":"ContainerStarted","Data":"475c8ace2634abe4804a687788dddefda337b0ede33a4523856c1a40e3faeb6b"} Nov 02 08:48:25.350947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.350915 5055 scope.go:122] "RemoveContainer" containerID="daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e" Nov 02 08:48:25.351428 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:48:25.351390 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e\": container with ID starting with daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e not found: ID does not exist" containerID="daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e" Nov 02 08:48:25.351530 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.351440 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e"} err="failed to get container status \"daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e\": rpc error: code = NotFound desc = could not find container \"daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e\": container with ID starting with daae78d042838ce0180291bbd6b301251972763afeb22d18e06b23f7e4b1cb0e not found: ID does not exist" Nov 02 08:48:25.380368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.380312 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:25.387400 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.387336 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:25.493065 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.493017 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv"] Nov 02 08:48:25.538972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.538929 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-gb7jv"] Nov 02 08:48:25.604436 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:25.604332 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podStartSLOduration=1.604317131 podStartE2EDuration="1.604317131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:48:25.603056165 +0000 UTC m=+2552.693676584" watchObservedRunningTime="2026-11-02 08:48:25.604317131 +0000 UTC m=+2552.694937544" Nov 02 08:48:26.348086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:26.348054 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:26.348966 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:26.348946 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:48:27.091974 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:27.091937 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" path="/var/lib/kubelet/pods/4381ccf4-b9f8-407c-a053-1a6ac3a352fd/volumes" Nov 02 08:48:34.506567 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:34.506470 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h"] Nov 02 08:48:34.552300 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:34.552263 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d5a5a-lsd4h"] Nov 02 08:48:35.092348 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:35.092305 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bbb21d-41d6-4522-8e72-1b4c0b057611" path="/var/lib/kubelet/pods/29bbb21d-41d6-4522-8e72-1b4c0b057611/volumes" Nov 02 08:48:41.119560 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:41.119524 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q"] Nov 02 08:48:41.120538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:41.119778 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" containerID="cri-o://27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f" gracePeriod=30 Nov 02 08:48:42.219968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.219928 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr"] Nov 02 08:48:42.220472 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.220458 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" Nov 02 08:48:42.220519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.220473 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" Nov 02 08:48:42.220597 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.220588 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4381ccf4-b9f8-407c-a053-1a6ac3a352fd" containerName="istio-proxy" Nov 02 08:48:42.225589 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.225561 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.274118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.274087 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:48:42.303879 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.303847 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:48:42.304100 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.303856 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:48:42.304100 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.303855 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:48:42.304475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.304450 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4fg\" (UniqueName: \"kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.304634 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.304616 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.304685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.304666 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.323624 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.323594 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr"] Nov 02 08:48:42.372048 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.372011 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:48:42.405695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.405649 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.405916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.405720 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.405916 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.405816 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4fg\" (UniqueName: \"kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.406473 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.406440 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.408248 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.408224 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.458980 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.458951 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4fg\" (UniqueName: \"kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg\") pod \"maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.538864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.538775 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:42.907574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:42.907532 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr"] Nov 02 08:48:42.910226 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:48:42.910183 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677b8b2e_546a_41b6_8071_657251ae7192.slice/crio-3c24334dd8d9a6d6027285c40c111b8c3fb383a11def80958bc90b50c7e0809f WatchSource:0}: Error finding container 3c24334dd8d9a6d6027285c40c111b8c3fb383a11def80958bc90b50c7e0809f: Status 404 returned error can't find the container with id 3c24334dd8d9a6d6027285c40c111b8c3fb383a11def80958bc90b50c7e0809f Nov 02 08:48:43.477385 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:43.477331 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" event={"ID":"677b8b2e-546a-41b6-8071-657251ae7192","Type":"ContainerStarted","Data":"82b82da099d76bdef0a03ce59c4fbbd3dfc9e9f17d0c8bd6eb84355d983e84f1"} Nov 02 08:48:43.477776 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:43.477411 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" event={"ID":"677b8b2e-546a-41b6-8071-657251ae7192","Type":"ContainerStarted","Data":"3c24334dd8d9a6d6027285c40c111b8c3fb383a11def80958bc90b50c7e0809f"} Nov 02 08:48:43.583106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:43.583046 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" podStartSLOduration=1.583031138 podStartE2EDuration="1.583031138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:48:43.582197797 +0000 UTC m=+2570.672818215" watchObservedRunningTime="2026-11-02 08:48:43.583031138 +0000 UTC m=+2570.673651599" Nov 02 08:48:44.484670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:44.484634 5055 generic.go:356] "Generic (PLEG): container finished" podID="677b8b2e-546a-41b6-8071-657251ae7192" containerID="82b82da099d76bdef0a03ce59c4fbbd3dfc9e9f17d0c8bd6eb84355d983e84f1" exitCode=0 Nov 02 08:48:44.485201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:44.484720 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" event={"ID":"677b8b2e-546a-41b6-8071-657251ae7192","Type":"ContainerDied","Data":"82b82da099d76bdef0a03ce59c4fbbd3dfc9e9f17d0c8bd6eb84355d983e84f1"} Nov 02 08:48:45.670342 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.670316 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:45.743228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.743106 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca\") pod \"677b8b2e-546a-41b6-8071-657251ae7192\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " Nov 02 08:48:45.743228 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.743200 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg\" (UniqueName: \"kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg\") pod \"677b8b2e-546a-41b6-8071-657251ae7192\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " Nov 02 08:48:45.743488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.743305 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca\") pod \"677b8b2e-546a-41b6-8071-657251ae7192\" (UID: \"677b8b2e-546a-41b6-8071-657251ae7192\") " Nov 02 08:48:45.743609 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.743584 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca" pod "677b8b2e-546a-41b6-8071-657251ae7192" (UID: "677b8b2e-546a-41b6-8071-657251ae7192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:45.745445 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.745422 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg" pod "677b8b2e-546a-41b6-8071-657251ae7192" (UID: "677b8b2e-546a-41b6-8071-657251ae7192"). InnerVolumeSpecName "kube-api-access-cz4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:45.745554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.745490 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca" pod "677b8b2e-546a-41b6-8071-657251ae7192" (UID: "677b8b2e-546a-41b6-8071-657251ae7192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:48:45.844347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.844310 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:45.844347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.844342 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b8b2e-546a-41b6-8071-657251ae7192-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:45.844347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:45.844354 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz4fg\" (UniqueName: \"kubernetes.io/projected/677b8b2e-546a-41b6-8071-657251ae7192-kube-api-access-cz4fg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.371895 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.371865 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:48:46.451577 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451545 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451621 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451659 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451768 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451698 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451781 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451829 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451888 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.451955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.452214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451881 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:46.452214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.451897 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:46.452214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452029 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data\") pod \"b174008c-b394-4f4a-8b42-e0afda9e5150\" (UID: \"b174008c-b394-4f4a-8b42-e0afda9e5150\") " Nov 02 08:48:46.452214 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452125 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:46.452433 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452408 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:46.452481 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452444 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:46.452643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452620 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.452643 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452641 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.452853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452651 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174008c-b394-4f4a-8b42-e0afda9e5150-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.452853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452664 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.452853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.452680 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.454361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.454337 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:46.454361 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.454341 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:46.454546 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.454515 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:48:46.454613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.454564 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg" pod "b174008c-b394-4f4a-8b42-e0afda9e5150" (UID: "b174008c-b394-4f4a-8b42-e0afda9e5150"). InnerVolumeSpecName "kube-api-access-rblcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:46.510026 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.509988 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" Nov 02 08:48:46.510285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.509991 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr" event={"ID":"677b8b2e-546a-41b6-8071-657251ae7192","Type":"ContainerDied","Data":"3c24334dd8d9a6d6027285c40c111b8c3fb383a11def80958bc90b50c7e0809f"} Nov 02 08:48:46.510285 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.510189 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c24334dd8d9a6d6027285c40c111b8c3fb383a11def80958bc90b50c7e0809f" Nov 02 08:48:46.512281 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.512249 5055 generic.go:356] "Generic (PLEG): container finished" podID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerID="27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f" exitCode=0 Nov 02 08:48:46.512431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.512310 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" Nov 02 08:48:46.512431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.512328 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" event={"ID":"b174008c-b394-4f4a-8b42-e0afda9e5150","Type":"ContainerDied","Data":"27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f"} Nov 02 08:48:46.512431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.512386 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q" event={"ID":"b174008c-b394-4f4a-8b42-e0afda9e5150","Type":"ContainerDied","Data":"1f09d46909a5f0455cdf775a375c4ee0a64793cdc10afef585c988562f10526d"} Nov 02 08:48:46.512431 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.512418 5055 scope.go:122] "RemoveContainer" containerID="27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f" Nov 02 08:48:46.524236 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.524206 5055 scope.go:122] "RemoveContainer" containerID="8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44" Nov 02 08:48:46.534943 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.534919 5055 scope.go:122] "RemoveContainer" containerID="27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f" Nov 02 08:48:46.535296 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:48:46.535272 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f\": container with ID starting with 27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f not found: ID does not exist" containerID="27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f" Nov 02 08:48:46.535403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.535305 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f"} err="failed to get container status \"27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f\": rpc error: code = NotFound desc = could not find container \"27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f\": container with ID starting with 27fc5d3cb5fee13385afe77837d10deafd4de321b6e3701671cbe10ae05c3d5f not found: ID does not exist" Nov 02 08:48:46.535403 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.535325 5055 scope.go:122] "RemoveContainer" containerID="8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44" Nov 02 08:48:46.535664 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:48:46.535643 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44\": container with ID starting with 8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44 not found: ID does not exist" containerID="8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44" Nov 02 08:48:46.535753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.535676 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44"} err="failed to get container status \"8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44\": rpc error: code = NotFound desc = could not find container \"8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44\": container with ID starting with 8fe33b4415c813f68a8cf772fb4ee71749602a7aadd1e13dffb08db3eba93a44 not found: ID does not exist" Nov 02 08:48:46.554652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.554609 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.554652 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.554654 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.554925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.554672 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblcg\" (UniqueName: \"kubernetes.io/projected/b174008c-b394-4f4a-8b42-e0afda9e5150-kube-api-access-rblcg\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.554925 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.554687 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174008c-b394-4f4a-8b42-e0afda9e5150-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:46.592761 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.592718 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq"] Nov 02 08:48:46.601799 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.601766 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn"] Nov 02 08:48:46.608995 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.608968 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf"] Nov 02 08:48:46.687602 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.687573 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q"] Nov 02 08:48:46.690023 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:46.689997 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d5a5a-openshift-default-68b79576d9-m5f8q"] Nov 02 08:48:47.092549 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:47.092507 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" path="/var/lib/kubelet/pods/b174008c-b394-4f4a-8b42-e0afda9e5150/volumes" Nov 02 08:48:54.919206 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:48:54.919165 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:48:56.239132 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:56.239097 5055 scope.go:122] "RemoveContainer" containerID="13284d34f711246f447c4b4341cbbd3a556c170f1c90afad6c998ce0401ccac2" Nov 02 08:48:56.248813 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:56.248790 5055 scope.go:122] "RemoveContainer" containerID="b33f6581719d6ed0ba84ccadf1ee72d5ff4ee787f5fcfc758fbf6b8335d9fd55" Nov 02 08:48:56.257712 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:56.257673 5055 scope.go:122] "RemoveContainer" containerID="7e9f77faf5c0dea560e951401ec990fe014c1448e234ba7707995d9e44d7acc8" Nov 02 08:48:56.268536 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:48:56.268511 5055 scope.go:122] "RemoveContainer" containerID="020a10c098be9ae2ec7007fb7d63658651187b806e33392d02e45ae77caa8d63" Nov 02 08:49:00.620708 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:00.620665 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:49:01.615283 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:01.615236 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" containerID="cri-o://11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf" gracePeriod=45 Nov 02 08:49:01.623511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:01.623448 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" containerID="cri-o://07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8" gracePeriod=45 Nov 02 08:49:01.630533 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:01.630500 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" containerID="cri-o://b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7" gracePeriod=45 Nov 02 08:49:01.809201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:01.807585 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": dial tcp 10.132.0.90:8000: connect: connection refused" Nov 02 08:49:01.964673 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:01.964641 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:49:02.018185 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.017104 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:49:02.021777 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.021747 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs\") pod \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " Nov 02 08:49:02.021937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.021818 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm\") pod \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " Nov 02 08:49:02.021937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.021856 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq\" (UniqueName: \"kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq\") pod \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " Nov 02 08:49:02.021937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.021923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache\") pod \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " Nov 02 08:49:02.022179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.021953 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir\") pod \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " Nov 02 08:49:02.022179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.021985 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7\" (UniqueName: \"kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7\") pod \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " Nov 02 08:49:02.022179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022013 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home\") pod \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " Nov 02 08:49:02.022739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022512 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home\") pod \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " Nov 02 08:49:02.022739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022623 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir\") pod \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " Nov 02 08:49:02.022739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022665 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm\") pod \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\" (UID: \"a439f420-80fa-4c2d-9a5b-3e2f733233f6\") " Nov 02 08:49:02.022739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022695 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs\" (UniqueName: \"kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs\") pod \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " Nov 02 08:49:02.022739 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022730 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache\") pod \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\" (UID: \"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609\") " Nov 02 08:49:02.023082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.022938 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home" pod "a439f420-80fa-4c2d-9a5b-3e2f733233f6" (UID: "a439f420-80fa-4c2d-9a5b-3e2f733233f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.023166 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.023108 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache" pod "a439f420-80fa-4c2d-9a5b-3e2f733233f6" (UID: "a439f420-80fa-4c2d-9a5b-3e2f733233f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.023241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.023202 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.023241 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.023224 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.023354 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.023287 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir" pod "a439f420-80fa-4c2d-9a5b-3e2f733233f6" (UID: "a439f420-80fa-4c2d-9a5b-3e2f733233f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.023464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.023446 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home" pod "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" (UID: "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.023646 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.023628 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir" pod "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" (UID: "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.024242 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.024205 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache" pod "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" (UID: "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.026807 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.026780 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:49:02.026956 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.026931 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm" pod "a439f420-80fa-4c2d-9a5b-3e2f733233f6" (UID: "a439f420-80fa-4c2d-9a5b-3e2f733233f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.027275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.027252 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq" pod "a439f420-80fa-4c2d-9a5b-3e2f733233f6" (UID: "a439f420-80fa-4c2d-9a5b-3e2f733233f6"). InnerVolumeSpecName "kube-api-access-qtprq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:02.027375 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.027356 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs" pod "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" (UID: "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:02.027515 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.027490 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs" pod "a439f420-80fa-4c2d-9a5b-3e2f733233f6" (UID: "a439f420-80fa-4c2d-9a5b-3e2f733233f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:02.028694 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.028667 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7" pod "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" (UID: "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609"). InnerVolumeSpecName "kube-api-access-lw6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:02.028928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.028903 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm" pod "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" (UID: "554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.123746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.123712 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir\") pod \"bdde112c-cddd-479e-ada5-2535c5a9ea39\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " Nov 02 08:49:02.123935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.123763 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs\") pod \"bdde112c-cddd-479e-ada5-2535c5a9ea39\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " Nov 02 08:49:02.123935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.123798 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache\") pod \"bdde112c-cddd-479e-ada5-2535c5a9ea39\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " Nov 02 08:49:02.123935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.123825 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm\") pod \"bdde112c-cddd-479e-ada5-2535c5a9ea39\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " Nov 02 08:49:02.123935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.123888 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6\" (UniqueName: \"kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6\") pod \"bdde112c-cddd-479e-ada5-2535c5a9ea39\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " Nov 02 08:49:02.124205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124001 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home\") pod \"bdde112c-cddd-479e-ada5-2535c5a9ea39\" (UID: \"bdde112c-cddd-479e-ada5-2535c5a9ea39\") " Nov 02 08:49:02.124205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124000 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir" pod "bdde112c-cddd-479e-ada5-2535c5a9ea39" (UID: "bdde112c-cddd-479e-ada5-2535c5a9ea39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.124205 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124198 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache" pod "bdde112c-cddd-479e-ada5-2535c5a9ea39" (UID: "bdde112c-cddd-479e-ada5-2535c5a9ea39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.124416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124395 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124421 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6k7\" (UniqueName: \"kubernetes.io/projected/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-kube-api-access-lw6k7\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124435 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124449 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124462 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439f420-80fa-4c2d-9a5b-3e2f733233f6-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124476 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124489 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124502 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439f420-80fa-4c2d-9a5b-3e2f733233f6-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124514 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124528 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtprq\" (UniqueName: \"kubernetes.io/projected/a439f420-80fa-4c2d-9a5b-3e2f733233f6-kube-api-access-qtprq\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124544 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124558 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.124785 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.124668 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home" pod "bdde112c-cddd-479e-ada5-2535c5a9ea39" (UID: "bdde112c-cddd-479e-ada5-2535c5a9ea39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.130187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.127051 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6" pod "bdde112c-cddd-479e-ada5-2535c5a9ea39" (UID: "bdde112c-cddd-479e-ada5-2535c5a9ea39"). InnerVolumeSpecName "kube-api-access-wxfq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:02.130187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.127171 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs" pod "bdde112c-cddd-479e-ada5-2535c5a9ea39" (UID: "bdde112c-cddd-479e-ada5-2535c5a9ea39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:02.130187 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.127609 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm" pod "bdde112c-cddd-479e-ada5-2535c5a9ea39" (UID: "bdde112c-cddd-479e-ada5-2535c5a9ea39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:02.225179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.225132 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfq6\" (UniqueName: \"kubernetes.io/projected/bdde112c-cddd-479e-ada5-2535c5a9ea39-kube-api-access-wxfq6\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.225179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.225180 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.225413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.225194 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdde112c-cddd-479e-ada5-2535c5a9ea39-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.225413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.225208 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdde112c-cddd-479e-ada5-2535c5a9ea39-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:02.637062 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.637021 5055 generic.go:356] "Generic (PLEG): container finished" podID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerID="11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf" exitCode=0 Nov 02 08:49:02.637492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.637196 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" event={"ID":"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609","Type":"ContainerDied","Data":"11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf"} Nov 02 08:49:02.637492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.637247 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" event={"ID":"554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609","Type":"ContainerDied","Data":"db9ad8ea45821dfdbe958105c0c4e03466f5edba8b30782dbb5b0558b3a6e736"} Nov 02 08:49:02.637492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.637269 5055 scope.go:122] "RemoveContainer" containerID="11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf" Nov 02 08:49:02.637492 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.637349 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq" Nov 02 08:49:02.639783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.639584 5055 generic.go:356] "Generic (PLEG): container finished" podID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerID="b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7" exitCode=0 Nov 02 08:49:02.639783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.639641 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" Nov 02 08:49:02.639783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.639704 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" event={"ID":"bdde112c-cddd-479e-ada5-2535c5a9ea39","Type":"ContainerDied","Data":"b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7"} Nov 02 08:49:02.639783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.639747 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" event={"ID":"bdde112c-cddd-479e-ada5-2535c5a9ea39","Type":"ContainerDied","Data":"dbfbfb2eb41de212912e1d574406cfd8a8fc6888d104f478ae524361e0ec2e79"} Nov 02 08:49:02.642710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.641418 5055 generic.go:356] "Generic (PLEG): container finished" podID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerID="07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8" exitCode=0 Nov 02 08:49:02.642710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.641463 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" event={"ID":"a439f420-80fa-4c2d-9a5b-3e2f733233f6","Type":"ContainerDied","Data":"07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8"} Nov 02 08:49:02.642710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.641486 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" event={"ID":"a439f420-80fa-4c2d-9a5b-3e2f733233f6","Type":"ContainerDied","Data":"3679827a03b54b2756eb4f2b9784f5e9d87703d518adbc81be2f2f36774f6c46"} Nov 02 08:49:02.642710 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.641539 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn" Nov 02 08:49:02.652800 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.652778 5055 scope.go:122] "RemoveContainer" containerID="11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf" Nov 02 08:49:02.653101 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:49:02.653077 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf\": container with ID starting with 11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf not found: ID does not exist" containerID="11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf" Nov 02 08:49:02.653201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.653114 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf"} err="failed to get container status \"11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf\": rpc error: code = NotFound desc = could not find container \"11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf\": container with ID starting with 11fd55394181ccd4f776214e6ae477384a3ec907801b55f8d60dc8ed8bc2dfcf not found: ID does not exist" Nov 02 08:49:02.653201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.653131 5055 scope.go:122] "RemoveContainer" containerID="b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7" Nov 02 08:49:02.661938 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.661923 5055 scope.go:122] "RemoveContainer" containerID="b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7" Nov 02 08:49:02.662232 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:49:02.662207 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7\": container with ID starting with b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7 not found: ID does not exist" containerID="b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7" Nov 02 08:49:02.662293 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.662243 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7"} err="failed to get container status \"b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7\": rpc error: code = NotFound desc = could not find container \"b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7\": container with ID starting with b68b3506034230927b73abbeee12073fcc5dfef85f9a5832d6adfe88602be3c7 not found: ID does not exist" Nov 02 08:49:02.662293 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.662264 5055 scope.go:122] "RemoveContainer" containerID="07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8" Nov 02 08:49:02.670370 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.670354 5055 scope.go:122] "RemoveContainer" containerID="07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8" Nov 02 08:49:02.670612 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:49:02.670595 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8\": container with ID starting with 07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8 not found: ID does not exist" containerID="07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8" Nov 02 08:49:02.670676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.670617 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8"} err="failed to get container status \"07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8\": rpc error: code = NotFound desc = could not find container \"07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8\": container with ID starting with 07c96bbf96473c9a78d5def9d1ad0de0379afcf90e3b5db54c28fa0e3d64dad8 not found: ID does not exist" Nov 02 08:49:02.800164 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.798999 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf"] Nov 02 08:49:02.816055 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.815999 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf"] Nov 02 08:49:02.861439 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.861400 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-096685/simulator-w6-096685-kserve-66b84844cc-qb6tf" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:49:02.918326 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.918209 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq"] Nov 02 08:49:02.921829 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:02.921795 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-096685/distinct-w6-096685-kserve-9bbc79bc-jmzqq"] Nov 02 08:49:03.023306 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:03.023267 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn"] Nov 02 08:49:03.027554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:03.027527 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-096685/premium-w6-096685-kserve-fc57fb556-g58zn"] Nov 02 08:49:03.119441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:03.119404 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" path="/var/lib/kubelet/pods/554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609/volumes" Nov 02 08:49:03.120030 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:03.120012 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" path="/var/lib/kubelet/pods/a439f420-80fa-4c2d-9a5b-3e2f733233f6/volumes" Nov 02 08:49:03.120636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:03.120617 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" path="/var/lib/kubelet/pods/bdde112c-cddd-479e-ada5-2535c5a9ea39/volumes" Nov 02 08:49:07.928060 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928018 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8"] Nov 02 08:49:07.928571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928554 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928572 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928583 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928589 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928596 5055 cpu_manager.go:414] "containerMap: removing container" podUID="677b8b2e-546a-41b6-8071-657251ae7192" containerName="revoke-keys" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928601 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677b8b2e-546a-41b6-8071-657251ae7192" containerName="revoke-keys" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928609 5055 cpu_manager.go:414] "containerMap: removing container" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" Nov 02 08:49:07.928618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928614 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928622 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928627 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928635 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928640 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928711 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a439f420-80fa-4c2d-9a5b-3e2f733233f6" containerName="main" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928720 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928726 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="677b8b2e-546a-41b6-8071-657251ae7192" containerName="revoke-keys" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928732 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="554f3a1f-f6b4-4c38-a1d3-41ab2c3a2609" containerName="main" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928740 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b174008c-b394-4f4a-8b42-e0afda9e5150" containerName="istio-proxy" Nov 02 08:49:07.928873 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.928751 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdde112c-cddd-479e-ada5-2535c5a9ea39" containerName="main" Nov 02 08:49:07.933757 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.933735 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:07.982901 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.982859 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:07.983133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.982923 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:07.983133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:07.983021 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zml96\" (UniqueName: \"kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.033853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.033805 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:49:08.033853 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.033830 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:49:08.034068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.033871 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:49:08.034068 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.033955 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:49:08.038264 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.038236 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc"] Nov 02 08:49:08.042332 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.042306 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8"] Nov 02 08:49:08.042464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.042420 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.083393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.083356 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc"] Nov 02 08:49:08.083821 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.083797 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.083927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.083839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.083927 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.083913 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.084044 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.083996 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf89g\" (UniqueName: \"kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.084103 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.084067 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.084193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.084169 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zml96\" (UniqueName: \"kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.084750 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.084723 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.086331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.086313 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.151115 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.151082 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zml96\" (UniqueName: \"kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96\") pod \"maas-api-revoke-sub-d9a40845-8hzh8\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.185359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.185288 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.185359 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.185322 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.185613 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.185464 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf89g\" (UniqueName: \"kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.186022 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.185997 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.187654 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.187631 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.239276 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.239244 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf89g\" (UniqueName: \"kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g\") pod \"maas-api-revoke-sub-b49287f9-lhspc\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.247544 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.247483 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:08.358152 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.358097 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:08.626656 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.626629 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8"] Nov 02 08:49:08.628448 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:49:08.628413 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416d28f0_23d0_4e67_a00a_b6586c40a2dc.slice/crio-e53737674c84c54931ff4514f3f84bdeed265dbf18a88409fe09b3f291a2d5bb WatchSource:0}: Error finding container e53737674c84c54931ff4514f3f84bdeed265dbf18a88409fe09b3f291a2d5bb: Status 404 returned error can't find the container with id e53737674c84c54931ff4514f3f84bdeed265dbf18a88409fe09b3f291a2d5bb Nov 02 08:49:08.697718 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.697675 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" event={"ID":"416d28f0-23d0-4e67-a00a-b6586c40a2dc","Type":"ContainerStarted","Data":"e53737674c84c54931ff4514f3f84bdeed265dbf18a88409fe09b3f291a2d5bb"} Nov 02 08:49:08.733441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:08.733407 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc"] Nov 02 08:49:09.705999 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:09.705963 5055 generic.go:356] "Generic (PLEG): container finished" podID="414c6209-b42d-45ed-ae72-542ce3352ac9" containerID="af84f11cad90317007aac4b54b2b6183bf1abc750ecd3361ad09b2f1956ef7b6" exitCode=0 Nov 02 08:49:09.706455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:09.706055 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" event={"ID":"414c6209-b42d-45ed-ae72-542ce3352ac9","Type":"ContainerDied","Data":"af84f11cad90317007aac4b54b2b6183bf1abc750ecd3361ad09b2f1956ef7b6"} Nov 02 08:49:09.706455 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:09.706111 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" event={"ID":"414c6209-b42d-45ed-ae72-542ce3352ac9","Type":"ContainerStarted","Data":"a515c9860ccc51db8dbef8f4c4ce6d9b0541fb38c46c0819987c85b6922a0e43"} Nov 02 08:49:09.707794 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:09.707763 5055 generic.go:356] "Generic (PLEG): container finished" podID="416d28f0-23d0-4e67-a00a-b6586c40a2dc" containerID="3a0dead0ff3d3b9bf5f30df0df87e8492f24cded27816ba0ceba7b096ac789e3" exitCode=0 Nov 02 08:49:09.707935 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:09.707800 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" event={"ID":"416d28f0-23d0-4e67-a00a-b6586c40a2dc","Type":"ContainerDied","Data":"3a0dead0ff3d3b9bf5f30df0df87e8492f24cded27816ba0ceba7b096ac789e3"} Nov 02 08:49:10.715989 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.715950 5055 generic.go:356] "Generic (PLEG): container finished" podID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerID="1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5" exitCode=0 Nov 02 08:49:10.716495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.716021 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" event={"ID":"c25f01e3-98c1-46ea-a1c8-236f011cc3a5","Type":"ContainerDied","Data":"1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5"} Nov 02 08:49:10.716495 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.716472 5055 scope.go:122] "RemoveContainer" containerID="1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5" Nov 02 08:49:10.717965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.717932 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:49:10.718057 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.718025 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:49:10.718104 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.718057 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 02 08:49:10.919049 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.918734 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:10.922805 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:10.922775 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:11.018235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018125 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca\") pod \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " Nov 02 08:49:11.018235 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018216 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g\" (UniqueName: \"kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g\") pod \"414c6209-b42d-45ed-ae72-542ce3352ac9\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " Nov 02 08:49:11.018449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018289 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96\" (UniqueName: \"kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96\") pod \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " Nov 02 08:49:11.018449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018333 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca\") pod \"414c6209-b42d-45ed-ae72-542ce3352ac9\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " Nov 02 08:49:11.018449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018372 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca\") pod \"414c6209-b42d-45ed-ae72-542ce3352ac9\" (UID: \"414c6209-b42d-45ed-ae72-542ce3352ac9\") " Nov 02 08:49:11.018449 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018434 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca\") pod \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\" (UID: \"416d28f0-23d0-4e67-a00a-b6586c40a2dc\") " Nov 02 08:49:11.018611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018516 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca" pod "416d28f0-23d0-4e67-a00a-b6586c40a2dc" (UID: "416d28f0-23d0-4e67-a00a-b6586c40a2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:11.018928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018890 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca" pod "414c6209-b42d-45ed-ae72-542ce3352ac9" (UID: "414c6209-b42d-45ed-ae72-542ce3352ac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:11.019079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.018958 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:11.020751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.020713 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca" pod "414c6209-b42d-45ed-ae72-542ce3352ac9" (UID: "414c6209-b42d-45ed-ae72-542ce3352ac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:11.020953 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.020927 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96" pod "416d28f0-23d0-4e67-a00a-b6586c40a2dc" (UID: "416d28f0-23d0-4e67-a00a-b6586c40a2dc"). InnerVolumeSpecName "kube-api-access-zml96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:11.021037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.020962 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g" pod "414c6209-b42d-45ed-ae72-542ce3352ac9" (UID: "414c6209-b42d-45ed-ae72-542ce3352ac9"). InnerVolumeSpecName "kube-api-access-mf89g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:11.021232 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.021210 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca" pod "416d28f0-23d0-4e67-a00a-b6586c40a2dc" (UID: "416d28f0-23d0-4e67-a00a-b6586c40a2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:11.120876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.120831 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:11.120876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.120873 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d28f0-23d0-4e67-a00a-b6586c40a2dc-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:11.121090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.120888 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf89g\" (UniqueName: \"kubernetes.io/projected/414c6209-b42d-45ed-ae72-542ce3352ac9-kube-api-access-mf89g\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:11.121090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.120902 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zml96\" (UniqueName: \"kubernetes.io/projected/416d28f0-23d0-4e67-a00a-b6586c40a2dc-kube-api-access-zml96\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:11.121090 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.120917 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414c6209-b42d-45ed-ae72-542ce3352ac9-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:11.348773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.348672 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:49:11.725738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.725701 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" Nov 02 08:49:11.726246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.725730 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49287f9-lhspc" event={"ID":"414c6209-b42d-45ed-ae72-542ce3352ac9","Type":"ContainerDied","Data":"a515c9860ccc51db8dbef8f4c4ce6d9b0541fb38c46c0819987c85b6922a0e43"} Nov 02 08:49:11.726246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.725790 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a515c9860ccc51db8dbef8f4c4ce6d9b0541fb38c46c0819987c85b6922a0e43" Nov 02 08:49:11.728679 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.728644 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" event={"ID":"c25f01e3-98c1-46ea-a1c8-236f011cc3a5","Type":"ContainerStarted","Data":"262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28"} Nov 02 08:49:11.731313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.731286 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" Nov 02 08:49:11.731313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.731295 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a40845-8hzh8" event={"ID":"416d28f0-23d0-4e67-a00a-b6586c40a2dc","Type":"ContainerDied","Data":"e53737674c84c54931ff4514f3f84bdeed265dbf18a88409fe09b3f291a2d5bb"} Nov 02 08:49:11.731475 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:11.731336 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e53737674c84c54931ff4514f3f84bdeed265dbf18a88409fe09b3f291a2d5bb" Nov 02 08:49:12.388168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:12.385640 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:49:12.388168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:12.385761 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 02 08:49:12.388168 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:12.385807 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 02 08:49:13.380061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:13.380018 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 02 08:49:13.380637 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:13.380079 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 02 08:49:14.385176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:14.384495 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 02 08:49:14.385176 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:14.384556 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 02 08:49:15.379868 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:15.379833 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 02 08:49:15.380059 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:15.379897 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 02 08:49:16.379682 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:16.379646 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 02 08:49:16.380231 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:16.379717 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 02 08:49:17.384584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:17.384553 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:49:17.385063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:17.385046 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:49:17.385912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:17.385886 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:49:24.920501 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:49:24.920455 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:49:25.046151 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:25.046092 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh"] Nov 02 08:49:25.092539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:25.092509 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d"] Nov 02 08:49:25.094756 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:25.094744 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df2580-ndgsh"] Nov 02 08:49:25.096713 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:25.096691 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d46e07b-jdb7d"] Nov 02 08:49:27.092020 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:27.091976 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f" path="/var/lib/kubelet/pods/a7a8f263-2f7f-4c60-b5e8-1e321cb4a30f/volumes" Nov 02 08:49:27.092629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:27.092607 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed55156-814c-4560-bffa-e2aa9168e440" path="/var/lib/kubelet/pods/aed55156-814c-4560-bffa-e2aa9168e440/volumes" Nov 02 08:49:29.955288 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:29.955255 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:49:30.428867 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.428763 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq"] Nov 02 08:49:30.429407 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.429386 5055 cpu_manager.go:414] "containerMap: removing container" podUID="416d28f0-23d0-4e67-a00a-b6586c40a2dc" containerName="revoke-keys" Nov 02 08:49:30.429491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.429405 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416d28f0-23d0-4e67-a00a-b6586c40a2dc" containerName="revoke-keys" Nov 02 08:49:30.429491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.429441 5055 cpu_manager.go:414] "containerMap: removing container" podUID="414c6209-b42d-45ed-ae72-542ce3352ac9" containerName="revoke-keys" Nov 02 08:49:30.429491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.429448 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414c6209-b42d-45ed-ae72-542ce3352ac9" containerName="revoke-keys" Nov 02 08:49:30.429620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.429561 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="414c6209-b42d-45ed-ae72-542ce3352ac9" containerName="revoke-keys" Nov 02 08:49:30.429620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.429581 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="416d28f0-23d0-4e67-a00a-b6586c40a2dc" containerName="revoke-keys" Nov 02 08:49:30.434538 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.434507 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.482887 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.482847 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:49:30.528914 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.528857 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:49:30.529227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.528922 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:49:30.529227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.528979 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:49:30.533899 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.533850 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq"] Nov 02 08:49:30.538554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.538513 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.538732 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.538570 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.538797 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.538763 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4sb\" (UniqueName: \"kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.640174 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.640092 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4sb\" (UniqueName: \"kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.640368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.640310 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.640368 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.640345 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.641075 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.641046 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.642754 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.642728 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.693313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.693232 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4sb\" (UniqueName: \"kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb\") pod \"maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:30.749311 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:30.749263 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:31.118811 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:31.118779 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq"] Nov 02 08:49:31.120972 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:49:31.120933 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1822e587_f692_46a3_99a4_2c9185a07206.slice/crio-657e2dc1af9d77f5dacdc3316527e3fc7a3c7aee0f773389d7117a6534b1eb19 WatchSource:0}: Error finding container 657e2dc1af9d77f5dacdc3316527e3fc7a3c7aee0f773389d7117a6534b1eb19: Status 404 returned error can't find the container with id 657e2dc1af9d77f5dacdc3316527e3fc7a3c7aee0f773389d7117a6534b1eb19 Nov 02 08:49:31.875648 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:31.875601 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" event={"ID":"1822e587-f692-46a3-99a4-2c9185a07206","Type":"ContainerStarted","Data":"2e66a8d00959286fac32a56d5eabd6570984dcfc86533cd036ea53e636740325"} Nov 02 08:49:31.875845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:31.875669 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" event={"ID":"1822e587-f692-46a3-99a4-2c9185a07206","Type":"ContainerStarted","Data":"657e2dc1af9d77f5dacdc3316527e3fc7a3c7aee0f773389d7117a6534b1eb19"} Nov 02 08:49:31.982109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:31.982053 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" podStartSLOduration=1.9820375019999998 podStartE2EDuration="1.982037502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:49:31.98122186 +0000 UTC m=+2619.071842278" watchObservedRunningTime="2026-11-02 08:49:31.982037502 +0000 UTC m=+2619.072657920" Nov 02 08:49:32.884447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:32.884407 5055 generic.go:356] "Generic (PLEG): container finished" podID="1822e587-f692-46a3-99a4-2c9185a07206" containerID="2e66a8d00959286fac32a56d5eabd6570984dcfc86533cd036ea53e636740325" exitCode=0 Nov 02 08:49:32.884864 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:32.884464 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" event={"ID":"1822e587-f692-46a3-99a4-2c9185a07206","Type":"ContainerDied","Data":"2e66a8d00959286fac32a56d5eabd6570984dcfc86533cd036ea53e636740325"} Nov 02 08:49:34.037295 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.037262 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:34.178503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.178420 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca\") pod \"1822e587-f692-46a3-99a4-2c9185a07206\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " Nov 02 08:49:34.178663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.178501 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca\") pod \"1822e587-f692-46a3-99a4-2c9185a07206\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " Nov 02 08:49:34.178663 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.178604 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb\" (UniqueName: \"kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb\") pod \"1822e587-f692-46a3-99a4-2c9185a07206\" (UID: \"1822e587-f692-46a3-99a4-2c9185a07206\") " Nov 02 08:49:34.179037 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.178932 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca" pod "1822e587-f692-46a3-99a4-2c9185a07206" (UID: "1822e587-f692-46a3-99a4-2c9185a07206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:34.179327 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.179177 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:34.180863 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.180839 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca" pod "1822e587-f692-46a3-99a4-2c9185a07206" (UID: "1822e587-f692-46a3-99a4-2c9185a07206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:34.182061 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.182041 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb" pod "1822e587-f692-46a3-99a4-2c9185a07206" (UID: "1822e587-f692-46a3-99a4-2c9185a07206"). InnerVolumeSpecName "kube-api-access-7w4sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:34.280363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.280329 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1822e587-f692-46a3-99a4-2c9185a07206-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:34.280363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.280359 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4sb\" (UniqueName: \"kubernetes.io/projected/1822e587-f692-46a3-99a4-2c9185a07206-kube-api-access-7w4sb\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:34.898851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.898816 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" Nov 02 08:49:34.898851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.898840 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq" event={"ID":"1822e587-f692-46a3-99a4-2c9185a07206","Type":"ContainerDied","Data":"657e2dc1af9d77f5dacdc3316527e3fc7a3c7aee0f773389d7117a6534b1eb19"} Nov 02 08:49:34.899157 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.898898 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="657e2dc1af9d77f5dacdc3316527e3fc7a3c7aee0f773389d7117a6534b1eb19" Nov 02 08:49:34.899247 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.899227 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr"] Nov 02 08:49:34.901795 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:34.901724 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-096685-s4wtr"] Nov 02 08:49:35.092406 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:35.092371 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677b8b2e-546a-41b6-8071-657251ae7192" path="/var/lib/kubelet/pods/677b8b2e-546a-41b6-8071-657251ae7192/volumes" Nov 02 08:49:48.275022 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:48.274980 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:49:54.923794 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:49:54.923748 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:49:56.320488 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:56.320458 5055 scope.go:122] "RemoveContainer" containerID="b86f94db2cb48b2ccca778583976e2111b9f7e14c32e893ce2d8b5a081487c59" Nov 02 08:49:56.330639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:56.330604 5055 scope.go:122] "RemoveContainer" containerID="34eb8dc58379adf04e4475ebe695b247ef1f1f15cf6ed73b7ed0c163bcb46cad" Nov 02 08:49:56.339620 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:49:56.339596 5055 scope.go:122] "RemoveContainer" containerID="0fb6c7af5987845b6df06c69708282532f8c6fd8e0066c7942d0150bb0d9ffa4" Nov 02 08:50:00.570896 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:00.570857 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq"] Nov 02 08:50:00.616688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:00.616646 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8f1e34-wlszq"] Nov 02 08:50:01.050347 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:01.050303 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b"] Nov 02 08:50:01.091069 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:01.091029 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1822e587-f692-46a3-99a4-2c9185a07206" path="/var/lib/kubelet/pods/1822e587-f692-46a3-99a4-2c9185a07206/volumes" Nov 02 08:50:01.095979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:01.095951 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb"] Nov 02 08:50:01.098162 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:01.098119 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f04fec-7l59b"] Nov 02 08:50:01.100124 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:01.100100 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3067018c-qlrkb"] Nov 02 08:50:03.091699 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:03.091663 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e096a55-e02e-4752-8a7e-477b2321b407" path="/var/lib/kubelet/pods/6e096a55-e02e-4752-8a7e-477b2321b407/volumes" Nov 02 08:50:03.092203 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:03.092097 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb" path="/var/lib/kubelet/pods/e978f39d-fcbf-4b92-a2b5-1eacdfe82fcb/volumes" Nov 02 08:50:04.694234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:04.694118 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:50:07.968364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:07.968328 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr"] Nov 02 08:50:07.968818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:07.968805 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1822e587-f692-46a3-99a4-2c9185a07206" containerName="revoke-keys" Nov 02 08:50:07.968866 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:07.968819 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1822e587-f692-46a3-99a4-2c9185a07206" containerName="revoke-keys" Nov 02 08:50:07.968941 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:07.968932 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1822e587-f692-46a3-99a4-2c9185a07206" containerName="revoke-keys" Nov 02 08:50:07.974148 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:07.974108 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.022804 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.022765 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:08.023013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.022969 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:50:08.069133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.069095 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:50:08.069133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.069100 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:50:08.071640 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.071613 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr"] Nov 02 08:50:08.154676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.154637 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.154676 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.154678 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.154961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.154716 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.256351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.256252 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.256351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.256298 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.256351 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.256348 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.256937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.256915 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.258738 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.258712 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.309878 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.309843 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr\") pod \"maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.586783 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.586694 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:08.962366 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:08.962332 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr"] Nov 02 08:50:08.963771 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:50:08.963743 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc2bba4_1226_4c2b_9620_38e9c775bd7d.slice/crio-a2393eec8a60730398c22c2eb6d7d8b2476eb403da84ff6c3cebc96277afe65a WatchSource:0}: Error finding container a2393eec8a60730398c22c2eb6d7d8b2476eb403da84ff6c3cebc96277afe65a: Status 404 returned error can't find the container with id a2393eec8a60730398c22c2eb6d7d8b2476eb403da84ff6c3cebc96277afe65a Nov 02 08:50:09.126596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:09.126556 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" event={"ID":"fcc2bba4-1226-4c2b-9620-38e9c775bd7d","Type":"ContainerStarted","Data":"a2393eec8a60730398c22c2eb6d7d8b2476eb403da84ff6c3cebc96277afe65a"} Nov 02 08:50:10.134968 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:10.134877 5055 generic.go:356] "Generic (PLEG): container finished" podID="fcc2bba4-1226-4c2b-9620-38e9c775bd7d" containerID="5a87ab43589b42b15775b5f31b0372a72b2f527bb5c076bbb70c07f09dd5cac9" exitCode=0 Nov 02 08:50:10.135364 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:10.134968 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" event={"ID":"fcc2bba4-1226-4c2b-9620-38e9c775bd7d","Type":"ContainerDied","Data":"5a87ab43589b42b15775b5f31b0372a72b2f527bb5c076bbb70c07f09dd5cac9"} Nov 02 08:50:11.292615 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.292584 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:11.394362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.394260 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca\") pod \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " Nov 02 08:50:11.394362 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.394350 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr\") pod \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " Nov 02 08:50:11.394633 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.394494 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca\") pod \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\" (UID: \"fcc2bba4-1226-4c2b-9620-38e9c775bd7d\") " Nov 02 08:50:11.394698 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.394681 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca" pod "fcc2bba4-1226-4c2b-9620-38e9c775bd7d" (UID: "fcc2bba4-1226-4c2b-9620-38e9c775bd7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:11.394876 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.394859 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:11.396700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.396672 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr" pod "fcc2bba4-1226-4c2b-9620-38e9c775bd7d" (UID: "fcc2bba4-1226-4c2b-9620-38e9c775bd7d"). InnerVolumeSpecName "kube-api-access-94csr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:11.396700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.396669 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca" pod "fcc2bba4-1226-4c2b-9620-38e9c775bd7d" (UID: "fcc2bba4-1226-4c2b-9620-38e9c775bd7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:11.495662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.495615 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-kube-api-access-94csr\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:11.495662 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:11.495655 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc2bba4-1226-4c2b-9620-38e9c775bd7d-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.149340 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:12.149311 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" Nov 02 08:50:12.149524 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:12.149341 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr" event={"ID":"fcc2bba4-1226-4c2b-9620-38e9c775bd7d","Type":"ContainerDied","Data":"a2393eec8a60730398c22c2eb6d7d8b2476eb403da84ff6c3cebc96277afe65a"} Nov 02 08:50:12.149524 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:12.149384 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2393eec8a60730398c22c2eb6d7d8b2476eb403da84ff6c3cebc96277afe65a" Nov 02 08:50:24.482596 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:24.482556 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr"] Nov 02 08:50:24.528377 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:24.528334 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bbb1eb-g9kvr"] Nov 02 08:50:24.921305 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:50:24.921227 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:50:25.073668 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:25.073626 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:50:25.092814 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:25.092776 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc2bba4-1226-4c2b-9620-38e9c775bd7d" path="/var/lib/kubelet/pods/fcc2bba4-1226-4c2b-9620-38e9c775bd7d/volumes" Nov 02 08:50:30.868639 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:30.868596 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm"] Nov 02 08:50:30.869208 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:30.868861 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" containerID="cri-o://262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28" gracePeriod=30 Nov 02 08:50:32.387979 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:32.387938 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:50:32.388413 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:32.387995 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:50:36.155635 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.155610 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:50:36.250086 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250049 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250116 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250239 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250266 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250294 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250357 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:36.250553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250373 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250418 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250469 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250501 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:36.250553 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250550 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo\") pod \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\" (UID: \"c25f01e3-98c1-46ea-a1c8-236f011cc3a5\") " Nov 02 08:50:36.250851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250618 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:36.250851 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250646 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:36.250947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.250859 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:36.251070 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.251048 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.251178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.251076 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.251178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.251092 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.251178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.251107 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.251178 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.251122 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.252579 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.252556 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "kube-api-access-j7ljh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:36.252685 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.252651 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:50:36.252972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.252946 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:36.252972 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.252950 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy" pod "c25f01e3-98c1-46ea-a1c8-236f011cc3a5" (UID: "c25f01e3-98c1-46ea-a1c8-236f011cc3a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:36.298490 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.298455 5055 generic.go:356] "Generic (PLEG): container finished" podID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerID="262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28" exitCode=0 Nov 02 08:50:36.298695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.298528 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" Nov 02 08:50:36.298695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.298546 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" event={"ID":"c25f01e3-98c1-46ea-a1c8-236f011cc3a5","Type":"ContainerDied","Data":"262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28"} Nov 02 08:50:36.298695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.298615 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm" event={"ID":"c25f01e3-98c1-46ea-a1c8-236f011cc3a5","Type":"ContainerDied","Data":"475c8ace2634abe4804a687788dddefda337b0ede33a4523856c1a40e3faeb6b"} Nov 02 08:50:36.298695 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.298628 5055 scope.go:122] "RemoveContainer" containerID="262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28" Nov 02 08:50:36.309079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.309054 5055 scope.go:122] "RemoveContainer" containerID="1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5" Nov 02 08:50:36.325539 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.325510 5055 scope.go:122] "RemoveContainer" containerID="262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28" Nov 02 08:50:36.325944 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:50:36.325912 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28\": container with ID starting with 262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28 not found: ID does not exist" containerID="262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28" Nov 02 08:50:36.326029 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.325958 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28"} err="failed to get container status \"262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28\": rpc error: code = NotFound desc = could not find container \"262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28\": container with ID starting with 262fa8792be953aa2756a7c8a9bf5ad3f1761ea060a89eb3d2713971594e1e28 not found: ID does not exist" Nov 02 08:50:36.326029 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.325985 5055 scope.go:122] "RemoveContainer" containerID="1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5" Nov 02 08:50:36.326462 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:50:36.326429 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5\": container with ID starting with 1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5 not found: ID does not exist" containerID="1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5" Nov 02 08:50:36.326611 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.326474 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5"} err="failed to get container status \"1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5\": rpc error: code = NotFound desc = could not find container \"1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5\": container with ID starting with 1d857abc765261f494a63f453a611c0c2076eedc83f4ad597be0220c32380fc5 not found: ID does not exist" Nov 02 08:50:36.351858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.351820 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ljh\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-kube-api-access-j7ljh\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.351858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.351851 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.351858 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.351863 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.352109 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.351872 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25f01e3-98c1-46ea-a1c8-236f011cc3a5-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:36.418017 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.417926 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs"] Nov 02 08:50:36.480860 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.480825 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm"] Nov 02 08:50:36.487302 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:36.487270 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbb1eb-openshift-default-6b6876488f-lzlnm"] Nov 02 08:50:37.090951 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:37.090916 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" path="/var/lib/kubelet/pods/c25f01e3-98c1-46ea-a1c8-236f011cc3a5/volumes" Nov 02 08:50:43.380234 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:43.380187 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr"] Nov 02 08:50:48.381159 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:48.381079 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" podUID="8da911be-0240-403d-ab93-ff1edab16626" containerName="payload-processing" containerID="cri-o://ab0e54afc03e58a652919c6581702dea42c7b1d12dc4bffef821c4f91f8d137b" gracePeriod=25 Nov 02 08:50:50.590350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590309 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st"] Nov 02 08:50:50.590882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590860 5055 cpu_manager.go:414] "containerMap: removing container" podUID="fcc2bba4-1226-4c2b-9620-38e9c775bd7d" containerName="revoke-keys" Nov 02 08:50:50.590937 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590888 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc2bba4-1226-4c2b-9620-38e9c775bd7d" containerName="revoke-keys" Nov 02 08:50:50.590976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590937 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" Nov 02 08:50:50.590976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590947 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" Nov 02 08:50:50.590976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590964 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" Nov 02 08:50:50.590976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.590971 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" Nov 02 08:50:50.591108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.591095 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" Nov 02 08:50:50.591186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.591114 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25f01e3-98c1-46ea-a1c8-236f011cc3a5" containerName="istio-proxy" Nov 02 08:50:50.591186 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.591125 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc2bba4-1226-4c2b-9620-38e9c775bd7d" containerName="revoke-keys" Nov 02 08:50:50.595981 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.595950 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.606441 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.606405 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqcd\" (UniqueName: \"kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.606629 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.606468 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.606706 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.606611 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.652720 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.652681 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st"] Nov 02 08:50:50.691988 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.691947 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:50.692199 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.692020 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:50:50.692262 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.692206 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:50:50.692262 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.692209 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:50:50.707350 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.707315 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.707540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.707425 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqcd\" (UniqueName: \"kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.707540 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.707489 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.708083 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.708062 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.709888 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.709865 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.760419 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.760385 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqcd\" (UniqueName: \"kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd\") pod \"maas-api-revoke-sub-01d1c424-w69st\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:50.911607 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:50.911463 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:51.285976 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.285951 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st"] Nov 02 08:50:51.288226 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:50:51.288194 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbde31b5_c2e5_449e_baf4_d8b5c19a8c57.slice/crio-04d3ffce08de049ef2489b5e3a9b441b9fb17ed944b826a7443929ba7acdcc7c WatchSource:0}: Error finding container 04d3ffce08de049ef2489b5e3a9b441b9fb17ed944b826a7443929ba7acdcc7c: Status 404 returned error can't find the container with id 04d3ffce08de049ef2489b5e3a9b441b9fb17ed944b826a7443929ba7acdcc7c Nov 02 08:50:51.407096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.407053 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" event={"ID":"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57","Type":"ContainerStarted","Data":"04d3ffce08de049ef2489b5e3a9b441b9fb17ed944b826a7443929ba7acdcc7c"} Nov 02 08:50:51.439557 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.439506 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" containerID="cri-o://e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b" gracePeriod=45 Nov 02 08:50:51.463262 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.463229 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused" Nov 02 08:50:51.687063 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.687033 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:50:51.717134 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717097 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache\") pod \"95370416-e4b9-4d7f-afb2-f052a9a30690\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " Nov 02 08:50:51.717331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717268 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home\") pod \"95370416-e4b9-4d7f-afb2-f052a9a30690\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " Nov 02 08:50:51.717331 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717300 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs\" (UniqueName: \"kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs\") pod \"95370416-e4b9-4d7f-afb2-f052a9a30690\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " Nov 02 08:50:51.717447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717330 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir\") pod \"95370416-e4b9-4d7f-afb2-f052a9a30690\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " Nov 02 08:50:51.717447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717366 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm\") pod \"95370416-e4b9-4d7f-afb2-f052a9a30690\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " Nov 02 08:50:51.717447 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717391 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache" pod "95370416-e4b9-4d7f-afb2-f052a9a30690" (UID: "95370416-e4b9-4d7f-afb2-f052a9a30690"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.717606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717443 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv\" (UniqueName: \"kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv\") pod \"95370416-e4b9-4d7f-afb2-f052a9a30690\" (UID: \"95370416-e4b9-4d7f-afb2-f052a9a30690\") " Nov 02 08:50:51.717606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717574 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home" pod "95370416-e4b9-4d7f-afb2-f052a9a30690" (UID: "95370416-e4b9-4d7f-afb2-f052a9a30690"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.717606 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717587 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir" pod "95370416-e4b9-4d7f-afb2-f052a9a30690" (UID: "95370416-e4b9-4d7f-afb2-f052a9a30690"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.718243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717833 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-home\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.718243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717851 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-tmp-dir\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.718243 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.717865 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-model-cache\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.719966 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.719941 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs" pod "95370416-e4b9-4d7f-afb2-f052a9a30690" (UID: "95370416-e4b9-4d7f-afb2-f052a9a30690"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:51.720114 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.720059 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm" pod "95370416-e4b9-4d7f-afb2-f052a9a30690" (UID: "95370416-e4b9-4d7f-afb2-f052a9a30690"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.720265 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.720244 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv" pod "95370416-e4b9-4d7f-afb2-f052a9a30690" (UID: "95370416-e4b9-4d7f-afb2-f052a9a30690"). InnerVolumeSpecName "kube-api-access-gjscv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:51.818626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.818550 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95370416-e4b9-4d7f-afb2-f052a9a30690-tls-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.818626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.818585 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95370416-e4b9-4d7f-afb2-f052a9a30690-dshm\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.818626 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:51.818600 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjscv\" (UniqueName: \"kubernetes.io/projected/95370416-e4b9-4d7f-afb2-f052a9a30690-kube-api-access-gjscv\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:52.414906 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.414813 5055 generic.go:356] "Generic (PLEG): container finished" podID="bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" containerID="dc3299df111eb312bddf3ea3409fcdf624acc6b6868c7e57716e4c311386de2a" exitCode=0 Nov 02 08:50:52.415079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.414899 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" event={"ID":"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57","Type":"ContainerDied","Data":"dc3299df111eb312bddf3ea3409fcdf624acc6b6868c7e57716e4c311386de2a"} Nov 02 08:50:52.416233 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.416209 5055 generic.go:356] "Generic (PLEG): container finished" podID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerID="e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b" exitCode=0 Nov 02 08:50:52.416345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.416247 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" Nov 02 08:50:52.416345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.416273 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" event={"ID":"95370416-e4b9-4d7f-afb2-f052a9a30690","Type":"ContainerDied","Data":"e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b"} Nov 02 08:50:52.416345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.416301 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs" event={"ID":"95370416-e4b9-4d7f-afb2-f052a9a30690","Type":"ContainerDied","Data":"6b0c72455cfdf29bfd5dfa45933cfcd74cf563ed9445a19f3c5ebeff0fd82eab"} Nov 02 08:50:52.416345 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.416321 5055 scope.go:122] "RemoveContainer" containerID="e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b" Nov 02 08:50:52.426587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.426568 5055 scope.go:122] "RemoveContainer" containerID="e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b" Nov 02 08:50:52.426863 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:50:52.426844 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b\": container with ID starting with e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b not found: ID does not exist" containerID="e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b" Nov 02 08:50:52.426940 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.426874 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b"} err="failed to get container status \"e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b\": rpc error: code = NotFound desc = could not find container \"e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b\": container with ID starting with e95dd55abd86eb6ce7f380ae76acb0793ec5bbaec04c88a07f118aeadb4dfa3b not found: ID does not exist" Nov 02 08:50:52.626722 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.626680 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs"] Nov 02 08:50:52.630129 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:52.630100 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/simulator-w1-d0628e-kserve-6d4fdf89c7-vrpbs"] Nov 02 08:50:53.091339 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.091306 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" path="/var/lib/kubelet/pods/95370416-e4b9-4d7f-afb2-f052a9a30690/volumes" Nov 02 08:50:53.593967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.593940 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:53.636882 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.636847 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd\" (UniqueName: \"kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd\") pod \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " Nov 02 08:50:53.637053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.636908 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca\") pod \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " Nov 02 08:50:53.637053 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.637039 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca\") pod \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\" (UID: \"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57\") " Nov 02 08:50:53.637271 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.637249 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca" pod "bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" (UID: "bbde31b5-c2e5-449e-baf4-d8b5c19a8c57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:53.637427 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.637408 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-openshift-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:53.639082 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.639055 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca" pod "bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" (UID: "bbde31b5-c2e5-449e-baf4-d8b5c19a8c57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:53.639190 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.639112 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd" pod "bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" (UID: "bbde31b5-c2e5-449e-baf4-d8b5c19a8c57"). InnerVolumeSpecName "kube-api-access-9wqcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:53.738898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.738862 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqcd\" (UniqueName: \"kubernetes.io/projected/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-kube-api-access-9wqcd\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:53.738898 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.738893 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde31b5-c2e5-449e-baf4-d8b5c19a8c57-maas-api-xks-ca\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:53.877006 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.876974 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:50:53.877209 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:53.876974 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tspm9_438dc60b-6968-41c4-b974-cff36e50ab18/console-operator/1.log" Nov 02 08:50:54.433803 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:54.433773 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" Nov 02 08:50:54.434275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:54.433808 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d1c424-w69st" event={"ID":"bbde31b5-c2e5-449e-baf4-d8b5c19a8c57","Type":"ContainerDied","Data":"04d3ffce08de049ef2489b5e3a9b441b9fb17ed944b826a7443929ba7acdcc7c"} Nov 02 08:50:54.434275 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:54.433865 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d3ffce08de049ef2489b5e3a9b441b9fb17ed944b826a7443929ba7acdcc7c" Nov 02 08:50:54.922271 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:50:54.922224 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:50:56.386967 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:56.386933 5055 scope.go:122] "RemoveContainer" containerID="e80350a670b23fa6a1a2412e9a479c073893df12863d2a78c0090612138a8fec" Nov 02 08:50:56.396371 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:56.396344 5055 scope.go:122] "RemoveContainer" containerID="d3878ca1d191ec99b6651b5d20ccbf81119ef9f66ad837a44945756842be81ef" Nov 02 08:50:56.406894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:56.406868 5055 scope.go:122] "RemoveContainer" containerID="0b7ac96698fdea4738b2056751ce7476e76c443c7cbe2882b1e287ea016dfa3c" Nov 02 08:50:56.827566 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:50:56.827528 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" podUID="8da911be-0240-403d-ab93-ff1edab16626" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:51:06.827993 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:06.827943 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" podUID="8da911be-0240-403d-ab93-ff1edab16626" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:51:07.371313 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:07.371267 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:51:08.549751 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.549709 5055 generic.go:356] "Generic (PLEG): container finished" podID="8da911be-0240-403d-ab93-ff1edab16626" containerID="ab0e54afc03e58a652919c6581702dea42c7b1d12dc4bffef821c4f91f8d137b" exitCode=0 Nov 02 08:51:08.550290 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.549787 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" event={"ID":"8da911be-0240-403d-ab93-ff1edab16626","Type":"ContainerDied","Data":"ab0e54afc03e58a652919c6581702dea42c7b1d12dc4bffef821c4f91f8d137b"} Nov 02 08:51:08.667965 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.667933 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:51:08.789050 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.789015 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume\") pod \"8da911be-0240-403d-ab93-ff1edab16626\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " Nov 02 08:51:08.789268 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.789221 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw\" (UniqueName: \"kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw\") pod \"8da911be-0240-403d-ab93-ff1edab16626\" (UID: \"8da911be-0240-403d-ab93-ff1edab16626\") " Nov 02 08:51:08.789417 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.789392 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume" pod "8da911be-0240-403d-ab93-ff1edab16626" (UID: "8da911be-0240-403d-ab93-ff1edab16626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:08.789516 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.789501 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da911be-0240-403d-ab93-ff1edab16626-plugins-config-volume\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:08.791464 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.791435 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw" pod "8da911be-0240-403d-ab93-ff1edab16626" (UID: "8da911be-0240-403d-ab93-ff1edab16626"). InnerVolumeSpecName "kube-api-access-j6szw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:08.891011 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:08.890970 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6szw\" (UniqueName: \"kubernetes.io/projected/8da911be-0240-403d-ab93-ff1edab16626-kube-api-access-j6szw\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:09.561084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:09.561053 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" Nov 02 08:51:09.561084 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:09.561069 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr" event={"ID":"8da911be-0240-403d-ab93-ff1edab16626","Type":"ContainerDied","Data":"61581ed810cf5634a5112f84287f6515da4ded9f39f00776aa99c36ca0bb3dff"} Nov 02 08:51:09.561636 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:09.561158 5055 scope.go:122] "RemoveContainer" containerID="ab0e54afc03e58a652919c6581702dea42c7b1d12dc4bffef821c4f91f8d137b" Nov 02 08:51:09.713670 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:09.713635 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr"] Nov 02 08:51:09.760088 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:09.760054 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d0628e-6c8bcd7b79-td4gr"] Nov 02 08:51:11.091619 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:11.091563 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da911be-0240-403d-ab93-ff1edab16626" path="/var/lib/kubelet/pods/8da911be-0240-403d-ab93-ff1edab16626/volumes" Nov 02 08:51:15.652382 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:15.652346 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8"] Nov 02 08:51:15.652835 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:15.652574 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" containerID="cri-o://1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617" gracePeriod=30 Nov 02 08:51:17.023959 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:17.023916 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69dbd95c57-6q7xh_915d8632-259a-4284-b10e-46ede9f3d67a/authorino/0.log" Nov 02 08:51:17.415796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:17.415711 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-x7d9h_e8a57647-59cb-49ba-a734-82752c851371/manager/0.log" Nov 02 08:51:17.678396 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:17.678324 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hqt9s_ca48b880-48ee-4744-8875-786ee567e6e5/manager/0.log" Nov 02 08:51:18.952246 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:18.952214 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2nmjm_cad0eb33-48f9-4d65-93ea-59f916a61471/manager/0.log" Nov 02 08:51:19.202961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:19.202881 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hfpsb_fd5f2f01-bb6b-4133-a218-4507aeb2a055/limitador/0.log" Nov 02 08:51:19.448391 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:19.448358 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2n2lb_1a2657e0-b801-4e94-858d-116a405bf9f7/manager/0.log" Nov 02 08:51:20.694398 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:20.694362 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hfpsb"] Nov 02 08:51:20.930192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:20.930064 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:51:21.025561 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025507 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.025789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025584 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.025789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025630 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.025789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025704 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.025789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025731 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.025789 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025766 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.026081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025805 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.026081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025849 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.026081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025876 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert\") pod \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\" (UID: \"d0a85325-9fcb-4782-98bb-09b5e92fc1db\") " Nov 02 08:51:21.026081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025877 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:51:21.026081 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.025889 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026195 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026220 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026349 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026366 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026376 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026386 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.026486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.026435 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:21.028344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.028309 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:51:21.028344 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.028338 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:51:21.028532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.028344 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:21.028532 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.028309 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr" pod "d0a85325-9fcb-4782-98bb-09b5e92fc1db" (UID: "d0a85325-9fcb-4782-98bb-09b5e92fc1db"). InnerVolumeSpecName "kube-api-access-gz6fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:21.126998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.126956 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6fr\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-kube-api-access-gz6fr\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.126998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.126996 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.126998 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.127008 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.127267 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.127017 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.127267 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.127026 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a85325-9fcb-4782-98bb-09b5e92fc1db-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:21.641363 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.641325 5055 generic.go:356] "Generic (PLEG): container finished" podID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerID="1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617" exitCode=0 Nov 02 08:51:21.641574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.641404 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" Nov 02 08:51:21.641574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.641407 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" event={"ID":"d0a85325-9fcb-4782-98bb-09b5e92fc1db","Type":"ContainerDied","Data":"1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617"} Nov 02 08:51:21.641574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.641491 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8" event={"ID":"d0a85325-9fcb-4782-98bb-09b5e92fc1db","Type":"ContainerDied","Data":"65490ec6838460683fbfc1ed91c3061dfe10dbfeb94f2a0cd1a553854d14b219"} Nov 02 08:51:21.641574 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.641516 5055 scope.go:122] "RemoveContainer" containerID="1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617" Nov 02 08:51:21.651512 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.651491 5055 scope.go:122] "RemoveContainer" containerID="3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261" Nov 02 08:51:21.660511 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.660492 5055 scope.go:122] "RemoveContainer" containerID="1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617" Nov 02 08:51:21.660776 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:51:21.660755 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617\": container with ID starting with 1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617 not found: ID does not exist" containerID="1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617" Nov 02 08:51:21.660848 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.660785 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617"} err="failed to get container status \"1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617\": rpc error: code = NotFound desc = could not find container \"1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617\": container with ID starting with 1a77b68e9df2981651734c749fdcfdd558b8160fea9010282f226d678212b617 not found: ID does not exist" Nov 02 08:51:21.660848 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.660804 5055 scope.go:122] "RemoveContainer" containerID="3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261" Nov 02 08:51:21.661069 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:51:21.661051 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261\": container with ID starting with 3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261 not found: ID does not exist" containerID="3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261" Nov 02 08:51:21.661123 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.661074 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261"} err="failed to get container status \"3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261\": rpc error: code = NotFound desc = could not find container \"3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261\": container with ID starting with 3d969b68668e893ff744bd5ba3578e82a7cb69aa7eb76c4c63aa44e97b5c2261 not found: ID does not exist" Nov 02 08:51:21.796616 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.796577 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8"] Nov 02 08:51:21.842841 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:21.842803 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-jlgg8"] Nov 02 08:51:23.091584 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:23.091546 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" path="/var/lib/kubelet/pods/d0a85325-9fcb-4782-98bb-09b5e92fc1db/volumes" Nov 02 08:51:24.918862 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:51:24.918825 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:51:32.315784 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:32.315746 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ccqh4_19187067-159b-4ddf-a72f-fe3ccc9a180f/manager/0.log" Nov 02 08:51:32.737593 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:32.737550 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dxvdj_23ba0863-68a4-4e22-895d-39cef92ac281/proxy/0.log" Nov 02 08:51:33.475644 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:33.475604 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69dbd95c57-6q7xh_915d8632-259a-4284-b10e-46ede9f3d67a/authorino/0.log" Nov 02 08:51:41.043955 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:41.043918 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp"] Nov 02 08:51:41.091093 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:41.091058 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11887a9-dr8tp"] Nov 02 08:51:43.090769 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:43.090725 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2ca0a7-15bc-4ff8-9586-99282ce41fd6" path="/var/lib/kubelet/pods/fd2ca0a7-15bc-4ff8-9586-99282ce41fd6/volumes" Nov 02 08:51:54.919344 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:51:54.919305 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:51:56.515414 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:56.515378 5055 scope.go:122] "RemoveContainer" containerID="697cc14ece152d75d188dc0d6290419c65973c818817bfea8bbbfab6d482f960" Nov 02 08:51:56.525112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:51:56.525086 5055 scope.go:122] "RemoveContainer" containerID="5e9818c5846fd8454c1d980b47669566366856490becc107d36b26b221562b59" Nov 02 08:52:05.435094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:05.435040 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ccqh4_19187067-159b-4ddf-a72f-fe3ccc9a180f/manager/0.log" Nov 02 08:52:05.856279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:05.856228 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dxvdj_23ba0863-68a4-4e22-895d-39cef92ac281/proxy/0.log" Nov 02 08:52:06.877486 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:06.877447 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-01d1c424-w69st_bbde31b5-c2e5-449e-baf4-d8b5c19a8c57/revoke-keys/0.log" Nov 02 08:52:07.109571 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:07.109542 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b49287f9-lhspc_414c6209-b42d-45ed-ae72-542ce3352ac9/revoke-keys/0.log" Nov 02 08:52:07.315165 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:07.315110 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d9a40845-8hzh8_416d28f0-23d0-4e67-a00a-b6586c40a2dc/revoke-keys/0.log" Nov 02 08:52:09.198211 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:09.198170 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69dbd95c57-6q7xh_915d8632-259a-4284-b10e-46ede9f3d67a/authorino/0.log" Nov 02 08:52:09.580256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:09.580223 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-x7d9h_e8a57647-59cb-49ba-a734-82752c851371/manager/0.log" Nov 02 08:52:09.813928 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:09.813893 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hqt9s_ca48b880-48ee-4744-8875-786ee567e6e5/manager/0.log" Nov 02 08:52:10.988416 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:10.988366 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2nmjm_cad0eb33-48f9-4d65-93ea-59f916a61471/manager/0.log" Nov 02 08:52:11.210558 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:11.210520 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hfpsb_fd5f2f01-bb6b-4133-a218-4507aeb2a055/limitador/0.log" Nov 02 08:52:11.434730 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:11.434648 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2n2lb_1a2657e0-b801-4e94-858d-116a405bf9f7/manager/0.log" Nov 02 08:52:12.175367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:12.175326 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5vd6w4_1d37579c-29e7-4813-bf47-42bd05875256/istio-proxy/0.log" Nov 02 08:52:13.215705 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:13.215672 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-kgvsk_53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed/istio-proxy/0.log" Nov 02 08:52:13.873806 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:13.873777 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-tskjl_d7de352b-640f-49b5-873b-1f8873d031b1/istio-proxy/1.log" Nov 02 08:52:14.101367 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:14.101329 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-ns25f_b662ed93-4c1c-4bd4-986f-68248f334b43/istio-proxy/0.log" Nov 02 08:52:14.770259 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:14.770219 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d5df879b9-lxqd6_6045c48b-d8e9-4c92-8201-3ccc48a38a91/router/0.log" Nov 02 08:52:15.247491 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:15.247440 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-tskjl_d7de352b-640f-49b5-873b-1f8873d031b1/istio-proxy/0.log" Nov 02 08:52:16.073154 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:16.073119 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2fdsg_702fd86a-7fd7-496c-ae16-105cc49fd781/main/0.log" Nov 02 08:52:16.342818 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:16.342732 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-w5vkv_71541873-6433-439a-86df-75e6f31f5c2c/main/0.log" Nov 02 08:52:17.096328 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:17.096281 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-lhp52_125943c9-4353-4684-80cd-f53ffe83a574/main/0.log" Nov 02 08:52:17.649503 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:17.649464 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-glz4k_5d13bf2a-d174-4b83-b363-ed42f963cff7/main/0.log" Nov 02 08:52:24.918792 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:52:24.918750 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:52:28.187897 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.187861 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cnbj7/must-gather-t2kwk"] Nov 02 08:52:28.188329 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188315 5055 cpu_manager.go:414] "containerMap: removing container" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" Nov 02 08:52:28.188376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188330 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" Nov 02 08:52:28.188376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188351 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" containerName="revoke-keys" Nov 02 08:52:28.188376 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188357 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" containerName="revoke-keys" Nov 02 08:52:28.188478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188380 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" Nov 02 08:52:28.188478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188385 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" Nov 02 08:52:28.188478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188400 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8da911be-0240-403d-ab93-ff1edab16626" containerName="payload-processing" Nov 02 08:52:28.188478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188404 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da911be-0240-403d-ab93-ff1edab16626" containerName="payload-processing" Nov 02 08:52:28.188478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188411 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" Nov 02 08:52:28.188478 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188415 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" Nov 02 08:52:28.188671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188480 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da911be-0240-403d-ab93-ff1edab16626" containerName="payload-processing" Nov 02 08:52:28.188671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188489 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" Nov 02 08:52:28.188671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188496 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="95370416-e4b9-4d7f-afb2-f052a9a30690" containerName="main" Nov 02 08:52:28.188671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188505 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbde31b5-c2e5-449e-baf4-d8b5c19a8c57" containerName="revoke-keys" Nov 02 08:52:28.188671 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.188660 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a85325-9fcb-4782-98bb-09b5e92fc1db" containerName="istio-proxy" Nov 02 08:52:28.191688 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.191601 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.286079 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.286033 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1543b6b-cb79-42c0-b504-472e628c7152-must-gather-output\") pod \"must-gather-t2kwk\" (UID: \"c1543b6b-cb79-42c0-b504-472e628c7152\") " pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.286314 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.286107 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvv4\" (UniqueName: \"kubernetes.io/projected/c1543b6b-cb79-42c0-b504-472e628c7152-kube-api-access-whvv4\") pod \"must-gather-t2kwk\" (UID: \"c1543b6b-cb79-42c0-b504-472e628c7152\") " pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.286543 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.286519 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cnbj7\"/\"default-dockercfg-4rgxz\"" Nov 02 08:52:28.286661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.286527 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cnbj7\"/\"openshift-service-ca.crt\"" Nov 02 08:52:28.286661 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.286572 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cnbj7\"/\"kube-root-ca.crt\"" Nov 02 08:52:28.289179 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.289151 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cnbj7/must-gather-t2kwk"] Nov 02 08:52:28.387052 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.387020 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1543b6b-cb79-42c0-b504-472e628c7152-must-gather-output\") pod \"must-gather-t2kwk\" (UID: \"c1543b6b-cb79-42c0-b504-472e628c7152\") " pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.387261 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.387091 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvv4\" (UniqueName: \"kubernetes.io/projected/c1543b6b-cb79-42c0-b504-472e628c7152-kube-api-access-whvv4\") pod \"must-gather-t2kwk\" (UID: \"c1543b6b-cb79-42c0-b504-472e628c7152\") " pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.387425 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.387406 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1543b6b-cb79-42c0-b504-472e628c7152-must-gather-output\") pod \"must-gather-t2kwk\" (UID: \"c1543b6b-cb79-42c0-b504-472e628c7152\") " pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.442133 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.442063 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvv4\" (UniqueName: \"kubernetes.io/projected/c1543b6b-cb79-42c0-b504-472e628c7152-kube-api-access-whvv4\") pod \"must-gather-t2kwk\" (UID: \"c1543b6b-cb79-42c0-b504-472e628c7152\") " pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.505753 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.505712 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cnbj7/must-gather-t2kwk" Nov 02 08:52:28.872317 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.872285 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cnbj7/must-gather-t2kwk"] Nov 02 08:52:28.874760 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:52:28.874718 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1543b6b_cb79_42c0_b504_472e628c7152.slice/crio-8bad42f5e689ee07c567da674b1e2a3db4945db323d77c1f464f797d804159bd WatchSource:0}: Error finding container 8bad42f5e689ee07c567da674b1e2a3db4945db323d77c1f464f797d804159bd: Status 404 returned error can't find the container with id 8bad42f5e689ee07c567da674b1e2a3db4945db323d77c1f464f797d804159bd Nov 02 08:52:28.877116 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:28.877097 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:52:29.042746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:29.042712 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cnbj7/must-gather-t2kwk" event={"ID":"c1543b6b-cb79-42c0-b504-472e628c7152","Type":"ContainerStarted","Data":"8bad42f5e689ee07c567da674b1e2a3db4945db323d77c1f464f797d804159bd"} Nov 02 08:52:36.102002 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:36.101960 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cnbj7/must-gather-t2kwk" event={"ID":"c1543b6b-cb79-42c0-b504-472e628c7152","Type":"ContainerStarted","Data":"cd7388962d7de7cc90f3716bfbc91d0843d44de97e6187854a8b8b79835f9a3f"} Nov 02 08:52:36.501100 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:36.501063 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk"] Nov 02 08:52:36.501334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:36.501287 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" containerID="cri-o://b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a" gracePeriod=30 Nov 02 08:52:36.673839 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:36.673805 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2phd8_73d52e94-07b5-4a34-b8ec-74e17ad7d096/global-pull-secret-syncer/0.log" Nov 02 08:52:37.110346 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:37.110309 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cnbj7/must-gather-t2kwk" event={"ID":"c1543b6b-cb79-42c0-b504-472e628c7152","Type":"ContainerStarted","Data":"111ce2dace25aae0f9b8495830b293e7958d9fc4dd38a6ad95984f2440d04482"} Nov 02 08:52:37.219519 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:37.219464 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cnbj7/must-gather-t2kwk" podStartSLOduration=2.202169551 podStartE2EDuration="9.219451224s" totalImagesPullingTime="7.017281673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:52:37.217652223 +0000 UTC m=+2804.308272641" watchObservedRunningTime="2026-11-02 08:52:37.219451224 +0000 UTC m=+2804.310071642" Nov 02 08:52:38.393746 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:38.393714 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-86sd2_e38a56d4-3eca-4c4d-b7f9-fbd80403e692/connection-checker/0.log" Nov 02 08:52:38.810978 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:38.810951 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cbw8w_1d14a7ee-7798-4273-9e07-68f3ad95b363/connection-checker/0.log" Nov 02 08:52:39.222796 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:39.222770 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qmp75_8a0e92c0-8b2e-439a-b72c-db5193d1c6bd/connection-checker/0.log" Nov 02 08:52:40.151127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:40.151093 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hlm8n_6449c7f8-c8f4-4022-8384-a85070a6493c/konnectivity-agent/0.log" Nov 02 08:52:41.761279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.761252 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:52:41.842266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842178 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842223 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842266 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842244 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842382 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842565 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842545 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842568 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:52:41.842689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842580 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842626 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842689 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842659 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842877 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842690 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data\") pod \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\" (UID: \"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed\") " Nov 02 08:52:41.842920 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.842892 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:52:41.843119 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.843097 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-certs\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.843230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.843110 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:52:41.843230 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.843124 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istiod-ca-cert\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.843393 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.843370 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:52:41.843554 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.843529 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:52:41.844764 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.844737 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:52:41.844869 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.844794 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "kube-api-access-gnf9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:52:41.844961 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.844947 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:52:41.845122 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.845099 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo" pod "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" (UID: "53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:52:41.892485 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.892455 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-247.us-west-2.compute.internal_34d28ca7a91586006c95d24bde05028b/haproxy/0.log" Nov 02 08:52:41.943930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.943881 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-credential-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.943930 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.943923 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-envoy\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.944192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.943942 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-workload-socket\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.944192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.943956 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-data\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.944192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.943975 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-token\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.944192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.943990 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnf9l\" (UniqueName: \"kubernetes.io/projected/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-kube-api-access-gnf9l\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:41.944192 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:41.944005 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed-istio-podinfo\") on node \"ip-10-0-139-247.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:52:42.138973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.138878 5055 generic.go:356] "Generic (PLEG): container finished" podID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerID="b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a" exitCode=0 Nov 02 08:52:42.138973 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.138927 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" Nov 02 08:52:42.139227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.138926 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" event={"ID":"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed","Type":"ContainerDied","Data":"b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a"} Nov 02 08:52:42.139227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.139059 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk" event={"ID":"53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed","Type":"ContainerDied","Data":"caec3e703f4ea8f564416507abbb592ba90f95828c74327299e07707b9cda23a"} Nov 02 08:52:42.139227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.139088 5055 scope.go:122] "RemoveContainer" containerID="b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a" Nov 02 08:52:42.150227 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.150206 5055 scope.go:122] "RemoveContainer" containerID="b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a" Nov 02 08:52:42.150515 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:52:42.150491 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a\": container with ID starting with b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a not found: ID does not exist" containerID="b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a" Nov 02 08:52:42.150587 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.150528 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a"} err="failed to get container status \"b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a\": rpc error: code = NotFound desc = could not find container \"b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a\": container with ID starting with b138bfd20f9e50a234627f0779aad5da467766899035387987fa51822fb7d93a not found: ID does not exist" Nov 02 08:52:42.293067 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.293030 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk"] Nov 02 08:52:42.298618 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:42.298589 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kgvsk"] Nov 02 08:52:43.090885 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:43.090838 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" path="/var/lib/kubelet/pods/53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed/volumes" Nov 02 08:52:46.789201 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:46.789132 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69dbd95c57-6q7xh_915d8632-259a-4284-b10e-46ede9f3d67a/authorino/0.log" Nov 02 08:52:47.108717 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.108632 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-x7d9h_e8a57647-59cb-49ba-a734-82752c851371/manager/0.log" Nov 02 08:52:47.164334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.164298 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2"] Nov 02 08:52:47.164773 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.164759 5055 cpu_manager.go:414] "containerMap: removing container" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" Nov 02 08:52:47.164834 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.164775 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" Nov 02 08:52:47.164913 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.164896 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fd0ce5-7958-41dd-9ac2-1fd3bdf261ed" containerName="istio-proxy" Nov 02 08:52:47.169983 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.169961 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.187947 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.187919 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-proc\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.188108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.187958 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-podres\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.188108 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.187980 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76bn\" (UniqueName: \"kubernetes.io/projected/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-kube-api-access-z76bn\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.188298 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.188265 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-lib-modules\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.188336 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.188317 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-sys\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.218716 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.218683 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgtvq\"/\"kube-root-ca.crt\"" Nov 02 08:52:47.264645 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.264607 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgtvq\"/\"openshift-service-ca.crt\"" Nov 02 08:52:47.264894 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.264696 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgtvq\"/\"default-dockercfg-p95c4\"" Nov 02 08:52:47.268484 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.268451 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2"] Nov 02 08:52:47.288912 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.288881 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-lib-modules\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.288919 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-sys\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.288957 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-proc\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.288987 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-podres\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.289013 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76bn\" (UniqueName: \"kubernetes.io/projected/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-kube-api-access-z76bn\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.289028 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-lib-modules\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.289051 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-sys\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289112 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.289099 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-proc\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.289422 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.289121 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-podres\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.343715 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.343687 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76bn\" (UniqueName: \"kubernetes.io/projected/44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1-kube-api-access-z76bn\") pod \"perf-node-gather-daemonset-75xb2\" (UID: \"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.483334 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.483299 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:47.522127 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.522076 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hqt9s_ca48b880-48ee-4744-8875-786ee567e6e5/manager/0.log" Nov 02 08:52:47.851845 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:47.851816 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2"] Nov 02 08:52:47.854004 ip-10-0-139-247 kubenswrapper[5055]: W1102 08:52:47.853968 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod44d5ab50_eb42_46a6_a97e_15d8f3cc0bf1.slice/crio-c35c5e66b33c6aa03f6ed4397ef43680f93678ce09c8768d6e5e238f8acb8c34 WatchSource:0}: Error finding container c35c5e66b33c6aa03f6ed4397ef43680f93678ce09c8768d6e5e238f8acb8c34: Status 404 returned error can't find the container with id c35c5e66b33c6aa03f6ed4397ef43680f93678ce09c8768d6e5e238f8acb8c34 Nov 02 08:52:48.177056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:48.176965 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" event={"ID":"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1","Type":"ContainerStarted","Data":"4e350ebec1f9b698389896f51bcb103c25e20fb156ddaaf645a76bdbb52d30fc"} Nov 02 08:52:48.177056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:48.177016 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" event={"ID":"44d5ab50-eb42-46a6-a97e-15d8f3cc0bf1","Type":"ContainerStarted","Data":"c35c5e66b33c6aa03f6ed4397ef43680f93678ce09c8768d6e5e238f8acb8c34"} Nov 02 08:52:48.177056 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:48.177045 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:48.284193 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:48.284100 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" podStartSLOduration=1.284085225 podStartE2EDuration="1.284085225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:52:48.282397273 +0000 UTC m=+2815.373017694" watchObservedRunningTime="2026-11-02 08:52:48.284085225 +0000 UTC m=+2815.374705643" Nov 02 08:52:48.846033 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:48.846003 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2nmjm_cad0eb33-48f9-4d65-93ea-59f916a61471/manager/0.log" Nov 02 08:52:49.160219 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:49.160118 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hfpsb_fd5f2f01-bb6b-4133-a218-4507aeb2a055/limitador/0.log" Nov 02 08:52:49.597256 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:49.597216 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2n2lb_1a2657e0-b801-4e94-858d-116a405bf9f7/manager/0.log" Nov 02 08:52:54.194996 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:54.194960 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-75xb2" Nov 02 08:52:54.919944 ip-10-0-139-247 kubenswrapper[5055]: E1102 08:52:54.919905 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48rm2" containerName="extractor" Nov 02 08:52:55.004411 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:55.004382 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-srmdt_611a6955-050e-4d16-b301-aa943a2f980e/cluster-monitoring-operator/0.log" Nov 02 08:52:56.561700 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:56.561661 5055 scope.go:122] "RemoveContainer" containerID="69816e975fc3c4fc356bf4b10b00c233220935c031086bc8cbbf5c6555e3c881" Nov 02 08:52:56.571253 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:56.571228 5055 scope.go:122] "RemoveContainer" containerID="a90023f7c75b94cf32f7aece6c1cc2678b02e76371fe11f74747be67aff2db92" Nov 02 08:52:56.580279 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:56.580251 5055 scope.go:122] "RemoveContainer" containerID="fbc751e67d6fefa41e1617a0697ec9503925dd39b25c150c2e7bbbc2437559fe" Nov 02 08:52:56.589096 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:56.589075 5055 scope.go:122] "RemoveContainer" containerID="26de72cba38db6a14d51503c37e2b3a79a54a80ade45c421053653c7236dab70" Nov 02 08:52:58.844526 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:58.844492 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nj7wb_d6ffe883-92cf-4e01-a8fd-80d03b061b47/node-exporter/0.log" Nov 02 08:52:59.251653 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:59.251624 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nj7wb_d6ffe883-92cf-4e01-a8fd-80d03b061b47/kube-rbac-proxy/0.log" Nov 02 08:52:59.818106 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:52:59.818074 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nj7wb_d6ffe883-92cf-4e01-a8fd-80d03b061b47/init-textfile/0.log" Nov 02 08:53:00.836013 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:53:00.835976 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ccqh4_19187067-159b-4ddf-a72f-fe3ccc9a180f/manager/0.log" Nov 02 08:53:00.952319 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:53:00.952278 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dxvdj_23ba0863-68a4-4e22-895d-39cef92ac281/proxy/0.log" Nov 02 08:53:06.404094 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:53:06.404019 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-jkvkf_1a7ed96f-845a-44ac-915e-ff9d26cf2a6e/prometheus-operator-admission-webhook/0.log" Nov 02 08:53:06.720118 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:53:06.720079 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ccqh4_19187067-159b-4ddf-a72f-fe3ccc9a180f/manager/0.log" Nov 02 08:53:07.461703 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:53:07.461669 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dxvdj_23ba0863-68a4-4e22-895d-39cef92ac281/proxy/0.log" Nov 02 08:53:10.041714 ip-10-0-139-247 kubenswrapper[5055]: I1102 08:53:10.041669 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5vd6w4_1d37579c-29e7-4813-bf47-42bd05875256/istio-proxy/0.log"